[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:07:29 dist-test-slave-5nnp krb5kdc[19799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:07:29 dist-test-slave-5nnp krb5kdc[19799](info): set up 2 sockets
Jan 31 03:07:29 dist-test-slave-5nnp krb5kdc[19799](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260131 03:07:31.324539 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.998s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:07:31 dist-test-slave-5nnp krb5kdc[19799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:07:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:07:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:31.462219 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:42125
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:45795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:42125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:31.540812 19815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:31.541033 19815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:31.541067 19815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:31.542505 19815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:07:31.542577 19815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:31.542600 19815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:31.542619 19815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:07:31.542647 19815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:07:31.544278 19815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:45795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:42125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:42125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:31.544545 19815 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:31.544740 19815 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:31.547807 19821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:31.547775 19820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:31.547767 19823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:31.547878 19815 server_base.cc:1060] running on GCE node
I20260131 03:07:31.548234 19815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:31.548451 19815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:31.549598 19815 hybrid_clock.cc:648] HybridClock initialized: now 1769828851549578 us; error 30 us; skew 500 ppm
Jan 31 03:07:31 dist-test-slave-5nnp krb5kdc[19799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828851, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:31.552052 19815 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:07:31.552611 19815 webserver.cc:492] Webserver started at http://127.19.84.62:36911/ using document root <none> and password file <none>
I20260131 03:07:31.552821 19815 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:31.552881 19815 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:31.553001 19815 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:31.554010 19815 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "a73c7917deff4eceafc6309197214285"
format_stamp: "Formatted at 2026-01-31 03:07:31 on dist-test-slave-5nnp"
server_key: "4e212aa21d7b9d197178333d7b6d4f65"
server_key_iv: "b8adda09eee4f75173b36854ffbeedea"
server_key_version: "encryptionkey@0"
I20260131 03:07:31.554339 19815 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "a73c7917deff4eceafc6309197214285"
format_stamp: "Formatted at 2026-01-31 03:07:31 on dist-test-slave-5nnp"
server_key: "4e212aa21d7b9d197178333d7b6d4f65"
server_key_iv: "b8adda09eee4f75173b36854ffbeedea"
server_key_version: "encryptionkey@0"
I20260131 03:07:31.555676 19815 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:07:31.556455 19830 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:31.556665 19815 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:31.556757 19815 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "a73c7917deff4eceafc6309197214285"
format_stamp: "Formatted at 2026-01-31 03:07:31 on dist-test-slave-5nnp"
server_key: "4e212aa21d7b9d197178333d7b6d4f65"
server_key_iv: "b8adda09eee4f75173b36854ffbeedea"
server_key_version: "encryptionkey@0"
I20260131 03:07:31.556850 19815 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:31.571760 19815 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:31.573647 19815 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:31.573807 19815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:31.578044 19815 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:42125
I20260131 03:07:31.578094 19882 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:42125 every 8 connection(s)
I20260131 03:07:31.578584 19815 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:07:31.579253 19883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:31.581815 19883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a73c7917deff4eceafc6309197214285: Bootstrap starting.
I20260131 03:07:31.582407 19883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a73c7917deff4eceafc6309197214285: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:31.582670 19883 log.cc:826] T 00000000000000000000000000000000 P a73c7917deff4eceafc6309197214285: Log is configured to *not* fsync() on all Append() calls
I20260131 03:07:31.583513 19883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a73c7917deff4eceafc6309197214285: No bootstrap required, opened a new log
I20260131 03:07:31.585381 19883 raft_consensus.cc:359] T 00000000000000000000000000000000 P a73c7917deff4eceafc6309197214285 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73c7917deff4eceafc6309197214285" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42125 } }
I20260131 03:07:31.585536 19883 raft_consensus.cc:385] T 00000000000000000000000000000000 P a73c7917deff4eceafc6309197214285 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:31.585651 19883 raft_consensus.cc:740] T 00000000000000000000000000000000 P a73c7917deff4eceafc6309197214285 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a73c7917deff4eceafc6309197214285, State: Initialized, Role: FOLLOWER
I20260131 03:07:31.585822 19883 consensus_queue.cc:260] T 00000000000000000000000000000000 P a73c7917deff4eceafc6309197214285 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73c7917deff4eceafc6309197214285" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42125 } }
I20260131 03:07:31.585899 19883 raft_consensus.cc:399] T 00000000000000000000000000000000 P a73c7917deff4eceafc6309197214285 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:07:31.585945 19883 raft_consensus.cc:493] T 00000000000000000000000000000000 P a73c7917deff4eceafc6309197214285 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:07:31.585992 19883 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a73c7917deff4eceafc6309197214285 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:31.586539 19883 raft_consensus.cc:515] T 00000000000000000000000000000000 P a73c7917deff4eceafc6309197214285 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73c7917deff4eceafc6309197214285" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42125 } }
I20260131 03:07:31.586671 19883 leader_election.cc:304] T 00000000000000000000000000000000 P a73c7917deff4eceafc6309197214285 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a73c7917deff4eceafc6309197214285; no voters:
I20260131 03:07:31.586853 19883 leader_election.cc:290] T 00000000000000000000000000000000 P a73c7917deff4eceafc6309197214285 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:07:31.586918 19886 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a73c7917deff4eceafc6309197214285 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:07:31.587020 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 19815
I20260131 03:07:31.587066 19886 raft_consensus.cc:697] T 00000000000000000000000000000000 P a73c7917deff4eceafc6309197214285 [term 1 LEADER]: Becoming Leader. State: Replica: a73c7917deff4eceafc6309197214285, State: Running, Role: LEADER
I20260131 03:07:31.587112 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:07:31.587090 19883 sys_catalog.cc:565] T 00000000000000000000000000000000 P a73c7917deff4eceafc6309197214285 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:07:31.587157 19886 consensus_queue.cc:237] T 00000000000000000000000000000000 P a73c7917deff4eceafc6309197214285 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73c7917deff4eceafc6309197214285" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42125 } }
I20260131 03:07:31.587495 19886 sys_catalog.cc:455] T 00000000000000000000000000000000 P a73c7917deff4eceafc6309197214285 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a73c7917deff4eceafc6309197214285" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73c7917deff4eceafc6309197214285" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42125 } } }
I20260131 03:07:31.587553 19886 sys_catalog.cc:458] T 00000000000000000000000000000000 P a73c7917deff4eceafc6309197214285 [sys.catalog]: This master's current role is: LEADER
I20260131 03:07:31.587692 19886 sys_catalog.cc:455] T 00000000000000000000000000000000 P a73c7917deff4eceafc6309197214285 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a73c7917deff4eceafc6309197214285. Latest consensus state: current_term: 1 leader_uuid: "a73c7917deff4eceafc6309197214285" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73c7917deff4eceafc6309197214285" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42125 } } }
I20260131 03:07:31.587738 19886 sys_catalog.cc:458] T 00000000000000000000000000000000 P a73c7917deff4eceafc6309197214285 [sys.catalog]: This master's current role is: LEADER
I20260131 03:07:31.587878 19792 external_mini_cluster.cc:1468] Setting key 640b00883751b7335b5219175147654f
W20260131 03:07:31.589185 19901 catalog_manager.cc:1568] T 00000000000000000000000000000000 P a73c7917deff4eceafc6309197214285: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:07:31.591698 19901 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:07:31.591789 19891 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:07:31.591995 19891 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:07:31.597759 19891 catalog_manager.cc:1357] Generated new cluster ID: 6e2fd6469fc744e9a6b7067525795aac
I20260131 03:07:31.597958 19891 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Jan 31 03:07:31 dist-test-slave-5nnp krb5kdc[19799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:31.607131 19904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:31.588978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55968 (local address 127.19.84.62:42125)
0131 03:07:31.596570 (+ 7592us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:31.596581 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:31.596611 (+ 30us) server_negotiation.cc:408] Connection header received
0131 03:07:31.596735 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:31.596741 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:31.596792 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:31.596948 (+ 156us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:31.598650 (+ 1702us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:31.599475 (+ 825us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:31.600062 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:31.600304 (+ 242us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:31.603218 (+ 2914us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:31.603230 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:31.603230 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:31.603250 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:31.605507 (+ 2257us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:31.605918 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:31.605921 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:31.605921 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:31.605977 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:31.606180 (+ 203us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:31.606181 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:31.606181 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:31.606381 (+ 200us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:31.606530 (+ 149us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:31.606766 (+ 236us) server_negotiation.cc:300] Negotiation successful
0131 03:07:31.606909 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7548,"thread_start_us":80,"threads_started":1}
I20260131 03:07:31.610742 19891 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:07:31.611280 19891 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:07:31.623073 19891 catalog_manager.cc:6027] T 00000000000000000000000000000000 P a73c7917deff4eceafc6309197214285: Generated new TSK 0
I20260131 03:07:31.623281 19891 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:31.674760 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:42125
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:45795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:31.756036 19911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:31.756222 19911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:31.756245 19911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:31.757710 19911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:31.757758 19911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:31.757789 19911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:07:31.759505 19911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:45795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:42125
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:31.759701 19911 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:31.759850 19911 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:31.762648 19916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:31.762650 19919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:31.762645 19917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:31.762794 19911 server_base.cc:1060] running on GCE node
I20260131 03:07:31.763098 19911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:31.763314 19911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:31.764475 19911 hybrid_clock.cc:648] HybridClock initialized: now 1769828851764446 us; error 47 us; skew 500 ppm
Jan 31 03:07:31 dist-test-slave-5nnp krb5kdc[19799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828851, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:31.766824 19911 init.cc:377] Logged in from keytab as kudu/127.19.84.1@KRBTEST.COM (short username kudu)
I20260131 03:07:31.767423 19911 webserver.cc:492] Webserver started at http://127.19.84.1:41683/ using document root <none> and password file <none>
I20260131 03:07:31.767624 19911 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:31.767676 19911 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:31.767796 19911 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:31.768749 19911 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "359458b42637473bb6049e4ce8a79611"
format_stamp: "Formatted at 2026-01-31 03:07:31 on dist-test-slave-5nnp"
server_key: "216c296da30c0eaaa87d3081128bd45e"
server_key_iv: "f5153fb75c4551a8b46cc88c1adac01e"
server_key_version: "encryptionkey@0"
I20260131 03:07:31.769068 19911 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "359458b42637473bb6049e4ce8a79611"
format_stamp: "Formatted at 2026-01-31 03:07:31 on dist-test-slave-5nnp"
server_key: "216c296da30c0eaaa87d3081128bd45e"
server_key_iv: "f5153fb75c4551a8b46cc88c1adac01e"
server_key_version: "encryptionkey@0"
I20260131 03:07:31.770275 19911 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:07:31.770994 19926 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:31.771162 19911 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260131 03:07:31.771245 19911 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "359458b42637473bb6049e4ce8a79611"
format_stamp: "Formatted at 2026-01-31 03:07:31 on dist-test-slave-5nnp"
server_key: "216c296da30c0eaaa87d3081128bd45e"
server_key_iv: "f5153fb75c4551a8b46cc88c1adac01e"
server_key_version: "encryptionkey@0"
I20260131 03:07:31.771318 19911 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:31.789201 19911 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:31.790973 19911 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:31.791134 19911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:31.791425 19911 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:07:31.791795 19911 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:07:31.791849 19911 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:31.791886 19911 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:07:31.791929 19911 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:31.797952 19911 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:37317
I20260131 03:07:31.798007 20039 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:37317 every 8 connection(s)
I20260131 03:07:31.798331 19911 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
I20260131 03:07:31.799564 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 19911
I20260131 03:07:31.799652 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:07:31.799808 19792 external_mini_cluster.cc:1468] Setting key 0b4603478926248082571aab38a1fe74
Jan 31 03:07:31 dist-test-slave-5nnp krb5kdc[19799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828851, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.84.2@KRBTEST.COM; defaulting to no policy
I20260131 03:07:31.808490 19904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:31.799062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:48365 (local address 127.19.84.62:42125)
0131 03:07:31.799144 (+ 82us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:31.799146 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:31.799244 (+ 98us) server_negotiation.cc:408] Connection header received
0131 03:07:31.799965 (+ 721us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:31.799967 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:31.799977 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:31.800069 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:31.800983 (+ 914us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:31.801454 (+ 471us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:31.802132 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:31.802315 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:31.805015 (+ 2700us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:31.805020 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:31.805020 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:31.805029 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:31.806806 (+ 1777us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:31.807288 (+ 482us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:31.807289 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:31.807290 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:31.807325 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:31.807680 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:31.807681 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:31.807682 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:31.807830 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:31.807922 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:31.808211 (+ 289us) server_negotiation.cc:300] Negotiation successful
0131 03:07:31.808353 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260131 03:07:31.808512 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:31.799003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:42125 (local address 127.19.84.1:48365)
0131 03:07:31.799137 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:31.799166 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:31.799858 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:31.800123 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:31.800127 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:31.800440 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:31.800885 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:31.800890 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:31.801541 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:31.801542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:31.802031 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:31.802033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:31.802189 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:31.803141 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:31.803160 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:31.804911 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:31.806946 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:31.806960 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:31.806963 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:31.807204 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:31.807464 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:31.807466 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:31.807466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:31.807598 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:31.807911 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:31.807916 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:31.808069 (+ 153us) client_negotiation.cc:769] Sending connection context
0131 03:07:31.808162 (+ 93us) client_negotiation.cc:240] Negotiation successful
0131 03:07:31.808349 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
I20260131 03:07:31.809140 20040 heartbeater.cc:344] Connected to a master server at 127.19.84.62:42125
I20260131 03:07:31.809257 20040 heartbeater.cc:461] Registering TS with master...
I20260131 03:07:31.809496 20040 heartbeater.cc:507] Master 127.19.84.62:42125 requested a full tablet report, sending...
I20260131 03:07:31.810016 19847 ts_manager.cc:194] Registered new tserver with Master: 359458b42637473bb6049e4ce8a79611 (127.19.84.1:37317)
I20260131 03:07:31.810745 19847 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.1@KRBTEST.COM'} at 127.19.84.1:48365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:31.855437 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:42125
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:45795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:31.935881 20047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:31.936074 20047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:31.936100 20047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:31.937536 20047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:31.937587 20047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:31.937614 20047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:07:31.939186 20047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:45795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:42125
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:31.939465 20047 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:31.939651 20047 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:31.942209 20053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:31.942206 20055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:31.942204 20052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:31.942294 20047 server_base.cc:1060] running on GCE node
I20260131 03:07:31.942641 20047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:31.942891 20047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:31.944057 20047 hybrid_clock.cc:648] HybridClock initialized: now 1769828851944039 us; error 37 us; skew 500 ppm
Jan 31 03:07:31 dist-test-slave-5nnp krb5kdc[19799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828851, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:31.946798 20047 init.cc:377] Logged in from keytab as kudu/127.19.84.2@KRBTEST.COM (short username kudu)
I20260131 03:07:31.947485 20047 webserver.cc:492] Webserver started at http://127.19.84.2:37317/ using document root <none> and password file <none>
I20260131 03:07:31.947695 20047 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:31.947749 20047 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:31.947876 20047 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:31.948856 20047 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "faab3701fba94cb6a06eee6204ef5404"
format_stamp: "Formatted at 2026-01-31 03:07:31 on dist-test-slave-5nnp"
server_key: "a4d5c73ecc69960ffb84674b26510fce"
server_key_iv: "68ef7a7eddc7512a28700a882ef4478b"
server_key_version: "encryptionkey@0"
I20260131 03:07:31.949180 20047 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "faab3701fba94cb6a06eee6204ef5404"
format_stamp: "Formatted at 2026-01-31 03:07:31 on dist-test-slave-5nnp"
server_key: "a4d5c73ecc69960ffb84674b26510fce"
server_key_iv: "68ef7a7eddc7512a28700a882ef4478b"
server_key_version: "encryptionkey@0"
I20260131 03:07:31.950449 20047 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260131 03:07:31.951246 20062 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:31.951514 20047 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:07:31.951611 20047 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "faab3701fba94cb6a06eee6204ef5404"
format_stamp: "Formatted at 2026-01-31 03:07:31 on dist-test-slave-5nnp"
server_key: "a4d5c73ecc69960ffb84674b26510fce"
server_key_iv: "68ef7a7eddc7512a28700a882ef4478b"
server_key_version: "encryptionkey@0"
I20260131 03:07:31.951700 20047 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:31.962888 20047 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:31.964846 20047 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:31.964985 20047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:31.965201 20047 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:07:31.965495 20047 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:07:31.965544 20047 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:31.965590 20047 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:07:31.965613 20047 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:31.971781 20047 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:45505
I20260131 03:07:31.971827 20175 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:45505 every 8 connection(s)
I20260131 03:07:31.972136 20047 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
Jan 31 03:07:31 dist-test-slave-5nnp krb5kdc[19799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828851, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:31.980058 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 20047
I20260131 03:07:31.980165 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:07:31.980361 19792 external_mini_cluster.cc:1468] Setting key 8effed14e643bc25d1ae4d610c7b25e4
I20260131 03:07:31.981045 19904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:31.972873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:43721 (local address 127.19.84.62:42125)
0131 03:07:31.972959 (+ 86us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:31.972961 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:31.973099 (+ 138us) server_negotiation.cc:408] Connection header received
0131 03:07:31.973611 (+ 512us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:31.973612 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:31.973626 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:31.973693 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:31.974430 (+ 737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:31.975025 (+ 595us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:31.975639 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:31.975810 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:31.978196 (+ 2386us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:31.978200 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:31.978200 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:31.978207 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:31.979612 (+ 1405us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:31.980063 (+ 451us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:31.980064 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:31.980065 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:31.980096 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:31.980344 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:31.980346 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:31.980346 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:31.980490 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:31.980570 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:31.980850 (+ 280us) server_negotiation.cc:300] Negotiation successful
0131 03:07:31.980952 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260131 03:07:31.981184 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:31.972836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:42125 (local address 127.19.84.2:43721)
0131 03:07:31.972995 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:31.973023 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:31.973505 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:31.973758 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:31.973761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:31.974064 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:31.974335 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:31.974338 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:31.975120 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:31.975121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:31.975558 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:31.975560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:31.975709 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:31.976659 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:31.976676 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:31.978088 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:31.979722 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:31.979729 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:31.979732 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:31.979985 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:31.980158 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:31.980159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:31.980160 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:31.980282 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:31.980558 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:31.980563 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:31.980700 (+ 137us) client_negotiation.cc:769] Sending connection context
0131 03:07:31.980828 (+ 128us) client_negotiation.cc:240] Negotiation successful
0131 03:07:31.980970 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260131 03:07:31.982813 20176 heartbeater.cc:344] Connected to a master server at 127.19.84.62:42125
I20260131 03:07:31.982939 20176 heartbeater.cc:461] Registering TS with master...
I20260131 03:07:31.983203 20176 heartbeater.cc:507] Master 127.19.84.62:42125 requested a full tablet report, sending...
I20260131 03:07:31.983749 19847 ts_manager.cc:194] Registered new tserver with Master: faab3701fba94cb6a06eee6204ef5404 (127.19.84.2:45505)
I20260131 03:07:31.984172 19847 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.2@KRBTEST.COM'} at 127.19.84.2:43721
WARNING: no policy specified for kudu/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:32.031433 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:42125
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:45795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:32.114081 20183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:32.114291 20183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:32.114317 20183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:32.115918 20183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:32.115973 20183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:32.116003 20183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:07:32.117975 20183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:45795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:42125
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:32.118244 20183 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:32.118455 20183 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:32.121140 20188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:32.121145 20191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:32.121142 20189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:32.121454 20183 server_base.cc:1060] running on GCE node
I20260131 03:07:32.121627 20183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:32.121807 20183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:32.122975 20183 hybrid_clock.cc:648] HybridClock initialized: now 1769828852122966 us; error 38 us; skew 500 ppm
Jan 31 03:07:32 dist-test-slave-5nnp krb5kdc[19799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828852, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:32.125674 20183 init.cc:377] Logged in from keytab as kudu/127.19.84.3@KRBTEST.COM (short username kudu)
I20260131 03:07:32.126353 20183 webserver.cc:492] Webserver started at http://127.19.84.3:36081/ using document root <none> and password file <none>
I20260131 03:07:32.126559 20183 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:32.126617 20183 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:32.126746 20183 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:32.127811 20183 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "929eafa9be494dbebd84186217aac114"
format_stamp: "Formatted at 2026-01-31 03:07:32 on dist-test-slave-5nnp"
server_key: "d977a50d39cfc33e6810c0bd142527d2"
server_key_iv: "326a6e273b7506fc34e9aa23edf8262b"
server_key_version: "encryptionkey@0"
I20260131 03:07:32.128139 20183 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "929eafa9be494dbebd84186217aac114"
format_stamp: "Formatted at 2026-01-31 03:07:32 on dist-test-slave-5nnp"
server_key: "d977a50d39cfc33e6810c0bd142527d2"
server_key_iv: "326a6e273b7506fc34e9aa23edf8262b"
server_key_version: "encryptionkey@0"
I20260131 03:07:32.129446 20183 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:32.130215 20198 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:32.130457 20183 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:32.130537 20183 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "929eafa9be494dbebd84186217aac114"
format_stamp: "Formatted at 2026-01-31 03:07:32 on dist-test-slave-5nnp"
server_key: "d977a50d39cfc33e6810c0bd142527d2"
server_key_iv: "326a6e273b7506fc34e9aa23edf8262b"
server_key_version: "encryptionkey@0"
I20260131 03:07:32.130590 20183 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:32.150210 20183 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:32.152509 20183 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:32.152631 20183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:32.152843 20183 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:07:32.153174 20183 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:07:32.153210 20183 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:32.153234 20183 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:07:32.153278 20183 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:32.159931 20183 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:40543
I20260131 03:07:32.160007 20311 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:40543 every 8 connection(s)
I20260131 03:07:32.160339 20183 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
Jan 31 03:07:32 dist-test-slave-5nnp krb5kdc[19799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828852, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:32.166885 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 20183
I20260131 03:07:32.166986 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:07:32.167155 19792 external_mini_cluster.cc:1468] Setting key f35d8f2713e5e914423aea973e0f0df8
I20260131 03:07:32.170300 19904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:32.161223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:58417 (local address 127.19.84.62:42125)
0131 03:07:32.161292 (+ 69us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:32.161294 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:32.161495 (+ 201us) server_negotiation.cc:408] Connection header received
0131 03:07:32.162130 (+ 635us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:32.162132 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:32.162143 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:32.162207 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:32.163033 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:32.163610 (+ 577us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:32.164162 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:32.164288 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:32.166925 (+ 2637us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:32.166931 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:32.166931 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:32.166941 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:32.168926 (+ 1985us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:32.169356 (+ 430us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:32.169358 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:32.169358 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:32.169384 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:32.169621 (+ 237us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:32.169622 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:32.169623 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:32.169736 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:32.169810 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:32.170144 (+ 334us) server_negotiation.cc:300] Negotiation successful
0131 03:07:32.170230 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260131 03:07:32.170435 20314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:32.161155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:42125 (local address 127.19.84.3:58417)
0131 03:07:32.161363 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:32.161400 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:32.162001 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:32.162277 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:32.162281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:32.162628 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:32.162924 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:32.162927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:32.163685 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:32.163686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:32.164075 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:32.164076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:32.164189 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:32.165188 (+ 999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:32.165208 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:32.166775 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:32.169021 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:32.169028 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:32.169030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:32.169266 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:32.169453 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:32.169453 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:32.169454 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:32.169542 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:32.169862 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:32.169868 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:32.169997 (+ 129us) client_negotiation.cc:769] Sending connection context
0131 03:07:32.170107 (+ 110us) client_negotiation.cc:240] Negotiation successful
0131 03:07:32.170225 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":81,"threads_started":1}
I20260131 03:07:32.171051 20312 heartbeater.cc:344] Connected to a master server at 127.19.84.62:42125
I20260131 03:07:32.171187 20312 heartbeater.cc:461] Registering TS with master...
I20260131 03:07:32.171425 20312 heartbeater.cc:507] Master 127.19.84.62:42125 requested a full tablet report, sending...
I20260131 03:07:32.171937 19847 ts_manager.cc:194] Registered new tserver with Master: 929eafa9be494dbebd84186217aac114 (127.19.84.3:40543)
I20260131 03:07:32.172293 19847 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:58417
I20260131 03:07:32.179883 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Jan 31 03:07:32 dist-test-slave-5nnp krb5kdc[19799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828852, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 31 03:07:32 dist-test-slave-5nnp krb5kdc[19799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828852, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.84.1@KRBTEST.COM
I20260131 03:07:32.203244 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:32.195160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51576 (local address 127.19.84.1:37317)
0131 03:07:32.195455 (+ 295us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:32.195463 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:32.195481 (+ 18us) server_negotiation.cc:408] Connection header received
0131 03:07:32.195508 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:32.195510 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:32.195524 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:32.195593 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:32.196317 (+ 724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:32.197165 (+ 848us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:32.197761 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:32.197994 (+ 233us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:32.200069 (+ 2075us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:32.200079 (+ 10us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:32.200079 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:32.200087 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:32.201829 (+ 1742us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:32.202199 (+ 370us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:32.202201 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:32.202202 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:32.202238 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:32.202409 (+ 171us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:32.202411 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:32.202411 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:32.202712 (+ 301us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:32.202870 (+ 158us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:32.202977 (+ 107us) server_negotiation.cc:300] Negotiation successful
0131 03:07:32.203056 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":142,"threads_started":1}
W20260131 03:07:32.203706 19954 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51576
Jan 31 03:07:32 dist-test-slave-5nnp krb5kdc[19799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 31 03:07:32 dist-test-slave-5nnp krb5kdc[19799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.1@KRBTEST.COM
I20260131 03:07:32.226696 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:32.219232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51580 (local address 127.19.84.1:37317)
0131 03:07:32.219341 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:32.219343 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:32.219355 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:07:32.219469 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:32.219470 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:32.219482 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:32.219541 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:32.220272 (+ 731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:32.220793 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:32.221379 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:32.221628 (+ 249us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:32.223831 (+ 2203us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:32.223835 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:32.223835 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:32.223844 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:32.225370 (+ 1526us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:32.225797 (+ 427us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:32.225798 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:32.225798 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:32.225825 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:32.226065 (+ 240us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:32.226066 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:32.226066 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:32.226150 (+ 84us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:32.226235 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:32.226496 (+ 261us) server_negotiation.cc:300] Negotiation successful
0131 03:07:32.226620 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260131 03:07:32.227464 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 19911
I20260131 03:07:32.232733 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 20047
I20260131 03:07:32.239323 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 20183
I20260131 03:07:32.243878 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 19815
2026-01-31T03:07:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2943 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:07:32 dist-test-slave-5nnp krb5kdc[20326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:07:32 dist-test-slave-5nnp krb5kdc[20326](info): set up 2 sockets
Jan 31 03:07:32 dist-test-slave-5nnp krb5kdc[20326](info): commencing operation
krb5kdc: starting...
W20260131 03:07:34.272596 19792 mini_kdc.cc:121] Time spent starting KDC: real 2.008s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:07:34 dist-test-slave-5nnp krb5kdc[20326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:07:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:07:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:34.417863 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:36321
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:36321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:34.500797 20342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:34.501006 20342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:34.501030 20342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:34.502645 20342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:07:34.502707 20342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:34.502722 20342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:34.502735 20342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:07:34.502748 20342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:07:34.504572 20342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:36321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:36321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:34.504844 20342 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:34.505051 20342 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:34.508147 20347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:34.508214 20342 server_base.cc:1060] running on GCE node
W20260131 03:07:34.508226 20348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:34.508150 20350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:34.508574 20342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:34.508809 20342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:34.509959 20342 hybrid_clock.cc:648] HybridClock initialized: now 1769828854509945 us; error 34 us; skew 500 ppm
Jan 31 03:07:34 dist-test-slave-5nnp krb5kdc[20326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828854, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:34.512447 20342 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:07:34.513105 20342 webserver.cc:492] Webserver started at http://127.19.84.62:46667/ using document root <none> and password file <none>
I20260131 03:07:34.513271 20342 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:34.513336 20342 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:34.513487 20342 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:34.514495 20342 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "4b4baaca35294191beb81842138ba8ae"
format_stamp: "Formatted at 2026-01-31 03:07:34 on dist-test-slave-5nnp"
server_key: "46b552967ba874d9ae3ac3b5103c7e3f"
server_key_iv: "386aa412873771aad641be7f2f0f7dcf"
server_key_version: "encryptionkey@0"
I20260131 03:07:34.514843 20342 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "4b4baaca35294191beb81842138ba8ae"
format_stamp: "Formatted at 2026-01-31 03:07:34 on dist-test-slave-5nnp"
server_key: "46b552967ba874d9ae3ac3b5103c7e3f"
server_key_iv: "386aa412873771aad641be7f2f0f7dcf"
server_key_version: "encryptionkey@0"
I20260131 03:07:34.516243 20342 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:07:34.517084 20357 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:34.517307 20342 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:34.517407 20342 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "4b4baaca35294191beb81842138ba8ae"
format_stamp: "Formatted at 2026-01-31 03:07:34 on dist-test-slave-5nnp"
server_key: "46b552967ba874d9ae3ac3b5103c7e3f"
server_key_iv: "386aa412873771aad641be7f2f0f7dcf"
server_key_version: "encryptionkey@0"
I20260131 03:07:34.517496 20342 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:34.529109 20342 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:34.534873 20342 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:34.535043 20342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:34.539489 20342 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:36321
I20260131 03:07:34.539551 20409 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:36321 every 8 connection(s)
I20260131 03:07:34.539901 20342 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:07:34.540541 20410 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:34.542228 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 20342
I20260131 03:07:34.542326 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:07:34.542484 19792 external_mini_cluster.cc:1468] Setting key 6c9f78bc51825ef38410e99f3a165415
I20260131 03:07:34.543603 20410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b4baaca35294191beb81842138ba8ae: Bootstrap starting.
I20260131 03:07:34.544365 20410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b4baaca35294191beb81842138ba8ae: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:34.544661 20410 log.cc:826] T 00000000000000000000000000000000 P 4b4baaca35294191beb81842138ba8ae: Log is configured to *not* fsync() on all Append() calls
I20260131 03:07:34.545362 20410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b4baaca35294191beb81842138ba8ae: No bootstrap required, opened a new log
I20260131 03:07:34.546913 20410 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4b4baaca35294191beb81842138ba8ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4baaca35294191beb81842138ba8ae" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36321 } }
I20260131 03:07:34.547073 20410 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4b4baaca35294191beb81842138ba8ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:34.547104 20410 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4b4baaca35294191beb81842138ba8ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b4baaca35294191beb81842138ba8ae, State: Initialized, Role: FOLLOWER
I20260131 03:07:34.547216 20410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b4baaca35294191beb81842138ba8ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4baaca35294191beb81842138ba8ae" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36321 } }
I20260131 03:07:34.547300 20410 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4b4baaca35294191beb81842138ba8ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:07:34.547335 20410 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4b4baaca35294191beb81842138ba8ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:07:34.547425 20410 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4b4baaca35294191beb81842138ba8ae [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:34.548187 20410 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4b4baaca35294191beb81842138ba8ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4baaca35294191beb81842138ba8ae" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36321 } }
I20260131 03:07:34.548285 20410 leader_election.cc:304] T 00000000000000000000000000000000 P 4b4baaca35294191beb81842138ba8ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b4baaca35294191beb81842138ba8ae; no voters:
I20260131 03:07:34.548476 20410 leader_election.cc:290] T 00000000000000000000000000000000 P 4b4baaca35294191beb81842138ba8ae [CANDIDATE]: Term 1 election: Requested vote from peers
Jan 31 03:07:34 dist-test-slave-5nnp krb5kdc[20326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:34.548509 20415 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4b4baaca35294191beb81842138ba8ae [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:07:34.548637 20415 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4b4baaca35294191beb81842138ba8ae [term 1 LEADER]: Becoming Leader. State: Replica: 4b4baaca35294191beb81842138ba8ae, State: Running, Role: LEADER
I20260131 03:07:34.548751 20410 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4b4baaca35294191beb81842138ba8ae [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:07:34.548719 20415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b4baaca35294191beb81842138ba8ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4baaca35294191beb81842138ba8ae" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36321 } }
I20260131 03:07:34.549068 20415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b4baaca35294191beb81842138ba8ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b4baaca35294191beb81842138ba8ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4baaca35294191beb81842138ba8ae" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36321 } } }
I20260131 03:07:34.549136 20415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b4baaca35294191beb81842138ba8ae [sys.catalog]: This master's current role is: LEADER
I20260131 03:07:34.549244 20415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b4baaca35294191beb81842138ba8ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b4baaca35294191beb81842138ba8ae. Latest consensus state: current_term: 1 leader_uuid: "4b4baaca35294191beb81842138ba8ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4baaca35294191beb81842138ba8ae" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36321 } } }
I20260131 03:07:34.549306 20415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b4baaca35294191beb81842138ba8ae [sys.catalog]: This master's current role is: LEADER
I20260131 03:07:34.549458 20419 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:07:34.550215 20419 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:07:34.552098 20419 catalog_manager.cc:1357] Generated new cluster ID: a98fa97740a544aba430b0e7bbf0d3cb
I20260131 03:07:34.552157 20419 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:07:34.553225 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:34.543065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44670 (local address 127.19.84.62:36321)
0131 03:07:34.543218 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:34.543228 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:34.543248 (+ 20us) server_negotiation.cc:408] Connection header received
0131 03:07:34.543598 (+ 350us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:34.543604 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:34.543663 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:34.543794 (+ 131us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:34.544751 (+ 957us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:34.545553 (+ 802us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:34.546291 (+ 738us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:34.546500 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:34.549175 (+ 2675us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:34.549183 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:34.549184 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:34.549210 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:34.551644 (+ 2434us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:34.552130 (+ 486us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:34.552133 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:34.552134 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:34.552209 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:34.552396 (+ 187us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:34.552397 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:34.552398 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:34.552655 (+ 257us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:34.552813 (+ 158us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:34.552919 (+ 106us) server_negotiation.cc:300] Negotiation successful
0131 03:07:34.553051 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260131 03:07:34.588359 20419 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:07:34.588982 20419 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:07:34.593609 20419 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 4b4baaca35294191beb81842138ba8ae: Generated new TSK 0
I20260131 03:07:34.593871 20419 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:34.650938 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:36321
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:34.735734 20438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:34.735951 20438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:34.735991 20438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:34.737520 20438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:34.737602 20438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:34.737668 20438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:07:34.739508 20438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:36321
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:34.739795 20438 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:34.740012 20438 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:34.743276 20443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:34.743769 20446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:34.743804 20444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:34.743956 20438 server_base.cc:1060] running on GCE node
I20260131 03:07:34.744143 20438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:34.744385 20438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:34.745563 20438 hybrid_clock.cc:648] HybridClock initialized: now 1769828854745548 us; error 42 us; skew 500 ppm
Jan 31 03:07:34 dist-test-slave-5nnp krb5kdc[20326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828854, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:34.748270 20438 init.cc:377] Logged in from keytab as kudu/127.19.84.1@KRBTEST.COM (short username kudu)
I20260131 03:07:34.748900 20438 webserver.cc:492] Webserver started at http://127.19.84.1:40137/ using document root <none> and password file <none>
I20260131 03:07:34.749104 20438 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:34.749156 20438 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:34.749317 20438 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:34.750376 20438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "1af8dd1091db4983bffe39c4e3e03192"
format_stamp: "Formatted at 2026-01-31 03:07:34 on dist-test-slave-5nnp"
server_key: "ff56f184055cd9dd1c4d717ae9f39d61"
server_key_iv: "24233d27a0384d12598f323c1b6d6cbd"
server_key_version: "encryptionkey@0"
I20260131 03:07:34.750710 20438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "1af8dd1091db4983bffe39c4e3e03192"
format_stamp: "Formatted at 2026-01-31 03:07:34 on dist-test-slave-5nnp"
server_key: "ff56f184055cd9dd1c4d717ae9f39d61"
server_key_iv: "24233d27a0384d12598f323c1b6d6cbd"
server_key_version: "encryptionkey@0"
I20260131 03:07:34.752094 20438 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260131 03:07:34.753037 20453 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:34.753252 20438 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:34.753340 20438 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "1af8dd1091db4983bffe39c4e3e03192"
format_stamp: "Formatted at 2026-01-31 03:07:34 on dist-test-slave-5nnp"
server_key: "ff56f184055cd9dd1c4d717ae9f39d61"
server_key_iv: "24233d27a0384d12598f323c1b6d6cbd"
server_key_version: "encryptionkey@0"
I20260131 03:07:34.753427 20438 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:34.762682 20438 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:34.764987 20438 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:34.765312 20438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:34.765571 20438 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:07:34.765923 20438 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:07:34.765976 20438 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:34.766021 20438 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:07:34.766050 20438 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:34.771999 20438 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:42601
I20260131 03:07:34.772045 20566 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:42601 every 8 connection(s)
I20260131 03:07:34.772349 20438 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
I20260131 03:07:34.775974 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 20438
I20260131 03:07:34.776082 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:07:34.776276 19792 external_mini_cluster.cc:1468] Setting key d57cdbae2f76f3f736675b50c3d9b74b
Jan 31 03:07:34 dist-test-slave-5nnp krb5kdc[20326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828854, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:34.782245 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:34.773089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:39969 (local address 127.19.84.62:36321)
0131 03:07:34.773161 (+ 72us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:34.773163 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:34.773357 (+ 194us) server_negotiation.cc:408] Connection header received
0131 03:07:34.773931 (+ 574us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:34.773933 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:34.773942 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:34.774007 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:34.774801 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:34.775292 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:34.775965 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:34.776159 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:34.779014 (+ 2855us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:34.779018 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:34.779019 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:34.779026 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:34.780692 (+ 1666us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:34.781237 (+ 545us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:34.781238 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:34.781239 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:34.781261 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:34.781516 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:34.781517 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:34.781517 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:34.781633 (+ 116us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:34.781701 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:34.782081 (+ 380us) server_negotiation.cc:300] Negotiation successful
0131 03:07:34.782172 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260131 03:07:34.782430 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:34.773068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:36321 (local address 127.19.84.1:39969)
0131 03:07:34.773234 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:34.773268 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:34.773843 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:34.774089 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:34.774093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:34.774407 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:34.774703 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:34.774707 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:34.775364 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:34.775364 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:34.775863 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:34.775865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:34.776349 (+ 484us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:34.777052 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:34.777081 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:34.778895 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:34.780785 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:34.780793 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:34.780798 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:34.781153 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:34.781329 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:34.781330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:34.781331 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:34.781431 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:34.781709 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:34.781714 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:34.781897 (+ 183us) client_negotiation.cc:769] Sending connection context
0131 03:07:34.782044 (+ 147us) client_negotiation.cc:240] Negotiation successful
0131 03:07:34.782212 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":102,"threads_started":1}
I20260131 03:07:34.782950 20567 heartbeater.cc:344] Connected to a master server at 127.19.84.62:36321
I20260131 03:07:34.783066 20567 heartbeater.cc:461] Registering TS with master...
I20260131 03:07:34.783293 20567 heartbeater.cc:507] Master 127.19.84.62:36321 requested a full tablet report, sending...
I20260131 03:07:34.784018 20374 ts_manager.cc:194] Registered new tserver with Master: 1af8dd1091db4983bffe39c4e3e03192 (127.19.84.1:42601)
WARNING: no policy specified for kudu/127.19.84.2@KRBTEST.COM; defaulting to no policy
I20260131 03:07:34.785212 20374 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.1@KRBTEST.COM'} at 127.19.84.1:39969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:34.830904 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:36321
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:34.911247 20574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:34.911492 20574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:34.911549 20574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:34.913028 20574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:34.913110 20574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:34.913156 20574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:07:34.914892 20574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:36321
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:34.915179 20574 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:34.915407 20574 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:34.919164 20579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:34.919164 20582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:34.919227 20580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:34.919250 20574 server_base.cc:1060] running on GCE node
I20260131 03:07:34.919627 20574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:34.919881 20574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:34.921025 20574 hybrid_clock.cc:648] HybridClock initialized: now 1769828854921002 us; error 34 us; skew 500 ppm
Jan 31 03:07:34 dist-test-slave-5nnp krb5kdc[20326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828854, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:34.923664 20574 init.cc:377] Logged in from keytab as kudu/127.19.84.2@KRBTEST.COM (short username kudu)
I20260131 03:07:34.924253 20574 webserver.cc:492] Webserver started at http://127.19.84.2:36429/ using document root <none> and password file <none>
I20260131 03:07:34.924456 20574 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:34.924511 20574 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:34.924841 20574 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:34.925879 20574 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "18b99e795d814af08c72129d249e5916"
format_stamp: "Formatted at 2026-01-31 03:07:34 on dist-test-slave-5nnp"
server_key: "416830eb7830ec45bb1c774715f9ab6d"
server_key_iv: "56a6eab7182ab1eae3b09b7bb6fd711f"
server_key_version: "encryptionkey@0"
I20260131 03:07:34.926204 20574 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "18b99e795d814af08c72129d249e5916"
format_stamp: "Formatted at 2026-01-31 03:07:34 on dist-test-slave-5nnp"
server_key: "416830eb7830ec45bb1c774715f9ab6d"
server_key_iv: "56a6eab7182ab1eae3b09b7bb6fd711f"
server_key_version: "encryptionkey@0"
I20260131 03:07:34.927536 20574 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260131 03:07:34.928366 20589 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:34.928608 20574 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:34.928704 20574 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "18b99e795d814af08c72129d249e5916"
format_stamp: "Formatted at 2026-01-31 03:07:34 on dist-test-slave-5nnp"
server_key: "416830eb7830ec45bb1c774715f9ab6d"
server_key_iv: "56a6eab7182ab1eae3b09b7bb6fd711f"
server_key_version: "encryptionkey@0"
I20260131 03:07:34.928784 20574 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:34.938723 20574 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:34.940845 20574 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:34.940997 20574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:34.941233 20574 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:07:34.941589 20574 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:07:34.941639 20574 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:34.941672 20574 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:07:34.941690 20574 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:34.948129 20574 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:41237
I20260131 03:07:34.948197 20702 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:41237 every 8 connection(s)
I20260131 03:07:34.948488 20574 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
Jan 31 03:07:34 dist-test-slave-5nnp krb5kdc[20326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828854, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:34.955440 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 20574
I20260131 03:07:34.955543 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:07:34.955757 19792 external_mini_cluster.cc:1468] Setting key 6b421ac1521ac66f91365d6d3fd38147
I20260131 03:07:34.958729 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:34.949277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:51147 (local address 127.19.84.62:36321)
0131 03:07:34.949349 (+ 72us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:34.949351 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:34.949565 (+ 214us) server_negotiation.cc:408] Connection header received
0131 03:07:34.950147 (+ 582us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:34.950148 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:34.950159 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:34.950224 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:34.951257 (+ 1033us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:34.951837 (+ 580us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:34.952432 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:34.952561 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:34.955041 (+ 2480us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:34.955046 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:34.955047 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:34.955054 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:34.956823 (+ 1769us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:34.957406 (+ 583us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:34.957407 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:34.957408 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:34.957438 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:34.957677 (+ 239us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:34.957678 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:34.957678 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:34.957821 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:34.957902 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:34.958523 (+ 621us) server_negotiation.cc:300] Negotiation successful
0131 03:07:34.958636 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260131 03:07:34.958871 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:34.949226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:36321 (local address 127.19.84.2:51147)
0131 03:07:34.949417 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:34.949454 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:34.950023 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:34.950285 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:34.950289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:34.950823 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:34.951118 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:34.951121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:34.951950 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:34.951951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:34.952346 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:34.952348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:34.952724 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:34.953244 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:34.953268 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:34.954927 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:34.956962 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:34.956970 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:34.956974 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:34.957276 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:34.957500 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:34.957501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:34.957501 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:34.957607 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:34.957889 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:34.957895 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:34.958351 (+ 456us) client_negotiation.cc:769] Sending connection context
0131 03:07:34.958501 (+ 150us) client_negotiation.cc:240] Negotiation successful
0131 03:07:34.958664 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":80,"threads_started":1}
I20260131 03:07:34.959640 20703 heartbeater.cc:344] Connected to a master server at 127.19.84.62:36321
I20260131 03:07:34.959841 20703 heartbeater.cc:461] Registering TS with master...
I20260131 03:07:34.960111 20703 heartbeater.cc:507] Master 127.19.84.62:36321 requested a full tablet report, sending...
I20260131 03:07:34.960525 20374 ts_manager.cc:194] Registered new tserver with Master: 18b99e795d814af08c72129d249e5916 (127.19.84.2:41237)
I20260131 03:07:34.960947 20374 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.2@KRBTEST.COM'} at 127.19.84.2:51147
WARNING: no policy specified for kudu/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:35.008562 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:36321
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:35.093750 20710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:35.093938 20710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:35.093964 20710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:35.095733 20710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:35.095799 20710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:35.095830 20710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:07:35.097688 20710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:36321
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:35.097941 20710 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:35.098166 20710 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:35.101198 20718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:35.101348 20715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:35.101198 20716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:35.101850 20710 server_base.cc:1060] running on GCE node
I20260131 03:07:35.102012 20710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:35.102213 20710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:35.103441 20710 hybrid_clock.cc:648] HybridClock initialized: now 1769828855103407 us; error 46 us; skew 500 ppm
Jan 31 03:07:35 dist-test-slave-5nnp krb5kdc[20326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828855, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:35.106153 20710 init.cc:377] Logged in from keytab as kudu/127.19.84.3@KRBTEST.COM (short username kudu)
I20260131 03:07:35.106802 20710 webserver.cc:492] Webserver started at http://127.19.84.3:37595/ using document root <none> and password file <none>
I20260131 03:07:35.107033 20710 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:35.107076 20710 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:35.107164 20710 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:35.108160 20710 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "289c1997a4764445a603834638cbdf73"
format_stamp: "Formatted at 2026-01-31 03:07:35 on dist-test-slave-5nnp"
server_key: "0c76a8188acb9af57238e9269c60cd80"
server_key_iv: "48aca9f1de49713aeebd811a20b7b802"
server_key_version: "encryptionkey@0"
I20260131 03:07:35.108548 20710 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "289c1997a4764445a603834638cbdf73"
format_stamp: "Formatted at 2026-01-31 03:07:35 on dist-test-slave-5nnp"
server_key: "0c76a8188acb9af57238e9269c60cd80"
server_key_iv: "48aca9f1de49713aeebd811a20b7b802"
server_key_version: "encryptionkey@0"
I20260131 03:07:35.109741 20710 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260131 03:07:35.110641 20725 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:35.110877 20710 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:07:35.110965 20710 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "289c1997a4764445a603834638cbdf73"
format_stamp: "Formatted at 2026-01-31 03:07:35 on dist-test-slave-5nnp"
server_key: "0c76a8188acb9af57238e9269c60cd80"
server_key_iv: "48aca9f1de49713aeebd811a20b7b802"
server_key_version: "encryptionkey@0"
I20260131 03:07:35.111016 20710 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:35.135180 20710 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:35.137388 20710 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:35.137504 20710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:35.137696 20710 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:07:35.138067 20710 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:07:35.138123 20710 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:35.138165 20710 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:07:35.138193 20710 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:35.144380 20710 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:40539
I20260131 03:07:35.144464 20840 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:40539 every 8 connection(s)
I20260131 03:07:35.144785 20710 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
Jan 31 03:07:35 dist-test-slave-5nnp krb5kdc[20326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828855, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:35.153882 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 20710
I20260131 03:07:35.153992 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:07:35.154152 19792 external_mini_cluster.cc:1468] Setting key 265c8232a0e1b0df5812c30cb64ae7aa
I20260131 03:07:35.154358 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:35.145677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:54865 (local address 127.19.84.62:36321)
0131 03:07:35.145738 (+ 61us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:35.145740 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:35.145868 (+ 128us) server_negotiation.cc:408] Connection header received
0131 03:07:35.146454 (+ 586us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:35.146455 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:35.146467 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:35.146534 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:35.147370 (+ 836us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.147959 (+ 589us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:35.148520 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.148669 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:35.151261 (+ 2592us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:35.151266 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:35.151266 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:35.151274 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:35.152775 (+ 1501us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:35.153203 (+ 428us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:35.153205 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:35.153205 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:35.153231 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:35.153410 (+ 179us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:35.153411 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:35.153411 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:35.153508 (+ 97us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:35.153567 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:35.154100 (+ 533us) server_negotiation.cc:300] Negotiation successful
0131 03:07:35.154233 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260131 03:07:35.154423 20843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:35.145585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:36321 (local address 127.19.84.3:54865)
0131 03:07:35.145733 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:35.145767 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:35.146267 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:35.146617 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:35.146621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:35.146962 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:35.147248 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:35.147251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.148055 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:35.148057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:35.148444 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:35.148446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.148871 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:35.149631 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:35.149654 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:35.151166 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:35.152871 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:35.152877 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:35.152880 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:35.153097 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:35.153299 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:35.153299 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:35.153300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:35.153359 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:35.153646 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:35.153651 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:35.153872 (+ 221us) client_negotiation.cc:769] Sending connection context
0131 03:07:35.154023 (+ 151us) client_negotiation.cc:240] Negotiation successful
0131 03:07:35.154190 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":86,"threads_started":1}
I20260131 03:07:35.155164 20841 heartbeater.cc:344] Connected to a master server at 127.19.84.62:36321
I20260131 03:07:35.155320 20841 heartbeater.cc:461] Registering TS with master...
I20260131 03:07:35.155616 20841 heartbeater.cc:507] Master 127.19.84.62:36321 requested a full tablet report, sending...
I20260131 03:07:35.156003 20373 ts_manager.cc:194] Registered new tserver with Master: 289c1997a4764445a603834638cbdf73 (127.19.84.3:40539)
I20260131 03:07:35.156386 20373 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:54865
I20260131 03:07:35.166426 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:07:35.174453 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:35.168266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44684 (local address 127.19.84.62:36321)
0131 03:07:35.168330 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:35.168332 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:35.168344 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:07:35.168464 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:35.168465 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:35.168475 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:35.168523 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:35.169328 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.169841 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:35.170632 (+ 791us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.170802 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:35.171884 (+ 1082us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:35.171888 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:35.171888 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:35.171895 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:35.173283 (+ 1388us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:35.173633 (+ 350us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:35.173634 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:35.173634 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:35.173656 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:35.173920 (+ 264us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:35.173921 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:35.173921 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:35.174053 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:35.174149 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:35.174298 (+ 149us) server_negotiation.cc:300] Negotiation successful
0131 03:07:35.174378 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260131 03:07:35.177251 20373 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:07:35.178505 20373 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:07:35.183923 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:35.180811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:41237 (local address 127.0.0.1:54866)
0131 03:07:35.181129 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:35.181138 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:35.181180 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:35.181659 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:35.181660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:35.181664 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:07:35.181932 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:35.181935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.182884 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:35.182885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:35.183627 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:35.183629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.183702 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:35.183741 (+ 39us) client_negotiation.cc:769] Sending connection context
0131 03:07:35.183792 (+ 51us) client_negotiation.cc:240] Negotiation successful
0131 03:07:35.183840 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":89,"threads_started":1}
I20260131 03:07:35.184490 20852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:35.180805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:42601 (local address 127.0.0.1:56564)
0131 03:07:35.180982 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:35.181013 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:35.181095 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:35.181518 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:35.181521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:35.181525 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:07:35.181964 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:35.181967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.183421 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:35.183422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:35.184265 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:35.184268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.184349 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:35.184353 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:07:35.184386 (+ 33us) client_negotiation.cc:240] Negotiation successful
0131 03:07:35.184421 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":58,"threads_started":1}
I20260131 03:07:35.184816 20854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:35.180913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54866 (local address 127.19.84.2:41237)
0131 03:07:35.181360 (+ 447us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:35.181366 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:35.181376 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:07:35.181410 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:35.181411 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:35.181426 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:35.181597 (+ 171us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:07:35.182024 (+ 427us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.182797 (+ 773us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:35.184051 (+ 1254us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.184582 (+ 531us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:35.184646 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:35.184678 (+ 32us) server_negotiation.cc:300] Negotiation successful
0131 03:07:35.184721 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":101,"threads_started":1}
I20260131 03:07:35.185026 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:35.181092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56564 (local address 127.19.84.1:42601)
0131 03:07:35.181352 (+ 260us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:35.181360 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:35.181371 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:07:35.181414 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:35.181416 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:35.181429 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:35.181481 (+ 52us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:07:35.182055 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.183299 (+ 1244us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:35.184377 (+ 1078us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.184827 (+ 450us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:35.184861 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:35.184908 (+ 47us) server_negotiation.cc:300] Negotiation successful
0131 03:07:35.184953 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":163,"threads_started":1}
I20260131 03:07:35.185256 20853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:35.181014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:40539 (local address 127.0.0.1:40264)
0131 03:07:35.181236 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:35.181242 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:35.181297 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:35.181955 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:35.181956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:35.181958 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:07:35.182137 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:35.182139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.183502 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:35.183503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:35.184964 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:35.184972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.185054 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:35.185058 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:07:35.185152 (+ 94us) client_negotiation.cc:240] Negotiation successful
0131 03:07:35.185192 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":70,"threads_started":1}
I20260131 03:07:35.185436 20501 tablet_service.cc:1505] Processing CreateTablet for tablet b8a3af424ce2461f945750680ea83063 (DEFAULT_TABLE table=test-table [id=f1546864c41b4abd88bd06b6aa350d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:07:35.185436 20637 tablet_service.cc:1505] Processing CreateTablet for tablet b8a3af424ce2461f945750680ea83063 (DEFAULT_TABLE table=test-table [id=f1546864c41b4abd88bd06b6aa350d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:07:35.185760 20637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8a3af424ce2461f945750680ea83063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:35.185760 20501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8a3af424ce2461f945750680ea83063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:35.186004 20856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:35.181163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40264 (local address 127.19.84.3:40539)
0131 03:07:35.181729 (+ 566us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:35.181738 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:35.181748 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:07:35.181785 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:35.181787 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:35.181812 (+ 25us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:35.181880 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:07:35.182204 (+ 324us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.183414 (+ 1210us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:35.185074 (+ 1660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.185768 (+ 694us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:35.185857 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:35.185892 (+ 35us) server_negotiation.cc:300] Negotiation successful
0131 03:07:35.185941 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":86,"threads_started":1}
I20260131 03:07:35.186666 20774 tablet_service.cc:1505] Processing CreateTablet for tablet b8a3af424ce2461f945750680ea83063 (DEFAULT_TABLE table=test-table [id=f1546864c41b4abd88bd06b6aa350d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:07:35.186928 20774 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8a3af424ce2461f945750680ea83063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:35.188692 20858 tablet_bootstrap.cc:492] T b8a3af424ce2461f945750680ea83063 P 1af8dd1091db4983bffe39c4e3e03192: Bootstrap starting.
I20260131 03:07:35.189334 20860 tablet_bootstrap.cc:492] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73: Bootstrap starting.
I20260131 03:07:35.189514 20858 tablet_bootstrap.cc:654] T b8a3af424ce2461f945750680ea83063 P 1af8dd1091db4983bffe39c4e3e03192: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:35.189800 20858 log.cc:826] T b8a3af424ce2461f945750680ea83063 P 1af8dd1091db4983bffe39c4e3e03192: Log is configured to *not* fsync() on all Append() calls
I20260131 03:07:35.189950 20860 tablet_bootstrap.cc:654] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:35.190125 20859 tablet_bootstrap.cc:492] T b8a3af424ce2461f945750680ea83063 P 18b99e795d814af08c72129d249e5916: Bootstrap starting.
I20260131 03:07:35.190246 20860 log.cc:826] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73: Log is configured to *not* fsync() on all Append() calls
I20260131 03:07:35.190627 20858 tablet_bootstrap.cc:492] T b8a3af424ce2461f945750680ea83063 P 1af8dd1091db4983bffe39c4e3e03192: No bootstrap required, opened a new log
I20260131 03:07:35.190703 20858 ts_tablet_manager.cc:1403] T b8a3af424ce2461f945750680ea83063 P 1af8dd1091db4983bffe39c4e3e03192: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:07:35.190726 20859 tablet_bootstrap.cc:654] T b8a3af424ce2461f945750680ea83063 P 18b99e795d814af08c72129d249e5916: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:35.190914 20860 tablet_bootstrap.cc:492] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73: No bootstrap required, opened a new log
I20260131 03:07:35.190986 20860 ts_tablet_manager.cc:1403] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:07:35.191026 20859 log.cc:826] T b8a3af424ce2461f945750680ea83063 P 18b99e795d814af08c72129d249e5916: Log is configured to *not* fsync() on all Append() calls
I20260131 03:07:35.191794 20859 tablet_bootstrap.cc:492] T b8a3af424ce2461f945750680ea83063 P 18b99e795d814af08c72129d249e5916: No bootstrap required, opened a new log
I20260131 03:07:35.191861 20859 ts_tablet_manager.cc:1403] T b8a3af424ce2461f945750680ea83063 P 18b99e795d814af08c72129d249e5916: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:07:35.192310 20860 raft_consensus.cc:359] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b99e795d814af08c72129d249e5916" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 41237 } } peers { permanent_uuid: "1af8dd1091db4983bffe39c4e3e03192" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 42601 } } peers { permanent_uuid: "289c1997a4764445a603834638cbdf73" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 40539 } }
I20260131 03:07:35.192446 20860 raft_consensus.cc:385] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:35.192484 20860 raft_consensus.cc:740] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289c1997a4764445a603834638cbdf73, State: Initialized, Role: FOLLOWER
I20260131 03:07:35.192420 20858 raft_consensus.cc:359] T b8a3af424ce2461f945750680ea83063 P 1af8dd1091db4983bffe39c4e3e03192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b99e795d814af08c72129d249e5916" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 41237 } } peers { permanent_uuid: "1af8dd1091db4983bffe39c4e3e03192" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 42601 } } peers { permanent_uuid: "289c1997a4764445a603834638cbdf73" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 40539 } }
I20260131 03:07:35.192567 20858 raft_consensus.cc:385] T b8a3af424ce2461f945750680ea83063 P 1af8dd1091db4983bffe39c4e3e03192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:35.192607 20860 consensus_queue.cc:260] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b99e795d814af08c72129d249e5916" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 41237 } } peers { permanent_uuid: "1af8dd1091db4983bffe39c4e3e03192" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 42601 } } peers { permanent_uuid: "289c1997a4764445a603834638cbdf73" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 40539 } }
I20260131 03:07:35.192602 20858 raft_consensus.cc:740] T b8a3af424ce2461f945750680ea83063 P 1af8dd1091db4983bffe39c4e3e03192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af8dd1091db4983bffe39c4e3e03192, State: Initialized, Role: FOLLOWER
I20260131 03:07:35.192888 20860 ts_tablet_manager.cc:1434] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260131 03:07:35.192941 20858 consensus_queue.cc:260] T b8a3af424ce2461f945750680ea83063 P 1af8dd1091db4983bffe39c4e3e03192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b99e795d814af08c72129d249e5916" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 41237 } } peers { permanent_uuid: "1af8dd1091db4983bffe39c4e3e03192" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 42601 } } peers { permanent_uuid: "289c1997a4764445a603834638cbdf73" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 40539 } }
I20260131 03:07:35.193001 20841 heartbeater.cc:499] Master 127.19.84.62:36321 was elected leader, sending a full tablet report...
I20260131 03:07:35.193388 20858 ts_tablet_manager.cc:1434] T b8a3af424ce2461f945750680ea83063 P 1af8dd1091db4983bffe39c4e3e03192: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260131 03:07:35.193516 20567 heartbeater.cc:499] Master 127.19.84.62:36321 was elected leader, sending a full tablet report...
I20260131 03:07:35.193531 20859 raft_consensus.cc:359] T b8a3af424ce2461f945750680ea83063 P 18b99e795d814af08c72129d249e5916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b99e795d814af08c72129d249e5916" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 41237 } } peers { permanent_uuid: "1af8dd1091db4983bffe39c4e3e03192" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 42601 } } peers { permanent_uuid: "289c1997a4764445a603834638cbdf73" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 40539 } }
I20260131 03:07:35.193651 20859 raft_consensus.cc:385] T b8a3af424ce2461f945750680ea83063 P 18b99e795d814af08c72129d249e5916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:35.193682 20859 raft_consensus.cc:740] T b8a3af424ce2461f945750680ea83063 P 18b99e795d814af08c72129d249e5916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18b99e795d814af08c72129d249e5916, State: Initialized, Role: FOLLOWER
I20260131 03:07:35.193774 20859 consensus_queue.cc:260] T b8a3af424ce2461f945750680ea83063 P 18b99e795d814af08c72129d249e5916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b99e795d814af08c72129d249e5916" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 41237 } } peers { permanent_uuid: "1af8dd1091db4983bffe39c4e3e03192" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 42601 } } peers { permanent_uuid: "289c1997a4764445a603834638cbdf73" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 40539 } }
I20260131 03:07:35.194283 20703 heartbeater.cc:499] Master 127.19.84.62:36321 was elected leader, sending a full tablet report...
I20260131 03:07:35.194346 20859 ts_tablet_manager.cc:1434] T b8a3af424ce2461f945750680ea83063 P 18b99e795d814af08c72129d249e5916: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260131 03:07:35.199008 20704 tablet.cc:2378] T b8a3af424ce2461f945750680ea83063 P 18b99e795d814af08c72129d249e5916: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260131 03:07:35.273027 20568 tablet.cc:2378] T b8a3af424ce2461f945750680ea83063 P 1af8dd1091db4983bffe39c4e3e03192: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:07:35.395327 20864 raft_consensus.cc:493] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260131 03:07:35.395368 20842 tablet.cc:2378] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:07:35.395517 20864 raft_consensus.cc:515] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b99e795d814af08c72129d249e5916" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 41237 } } peers { permanent_uuid: "1af8dd1091db4983bffe39c4e3e03192" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 42601 } } peers { permanent_uuid: "289c1997a4764445a603834638cbdf73" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 40539 } }
I20260131 03:07:35.395870 20864 leader_election.cc:290] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18b99e795d814af08c72129d249e5916 (127.19.84.2:41237), 1af8dd1091db4983bffe39c4e3e03192 (127.19.84.1:42601)
I20260131 03:07:35.398064 20843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:35.396093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:42601 (local address 127.19.84.3:59761)
0131 03:07:35.396161 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:35.396175 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:35.396244 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:35.396402 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:35.396403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:35.396405 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:07:35.396609 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:35.396611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.397255 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:35.397256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:35.397829 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:35.397831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.397917 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:35.397922 (+ 5us) client_negotiation.cc:769] Sending connection context
0131 03:07:35.397953 (+ 31us) client_negotiation.cc:240] Negotiation successful
0131 03:07:35.397992 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260131 03:07:35.398296 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:35.396093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:41237 (local address 127.19.84.3:47845)
0131 03:07:35.396348 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:35.396357 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:35.396434 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:35.396575 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:35.396575 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:35.396577 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:07:35.396801 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:35.396803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.397454 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:35.397459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:35.398052 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:35.398054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.398147 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:35.398150 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:07:35.398182 (+ 32us) client_negotiation.cc:240] Negotiation successful
0131 03:07:35.398217 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":91,"thread_start_us":93,"threads_started":1}
I20260131 03:07:35.398545 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:35.396228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:59761 (local address 127.19.84.1:42601)
0131 03:07:35.396269 (+ 41us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:35.396272 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:35.396294 (+ 22us) server_negotiation.cc:408] Connection header received
0131 03:07:35.396327 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:35.396328 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:35.396338 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:35.396397 (+ 59us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:07:35.396689 (+ 292us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.397169 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:35.397915 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.398401 (+ 486us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:35.398409 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:35.398434 (+ 25us) server_negotiation.cc:300] Negotiation successful
0131 03:07:35.398476 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260131 03:07:35.398804 20854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:35.396171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:47845 (local address 127.19.84.2:41237)
0131 03:07:35.396229 (+ 58us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:35.396231 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:35.396419 (+ 188us) server_negotiation.cc:408] Connection header received
0131 03:07:35.396500 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:35.396501 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:35.396514 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:35.396566 (+ 52us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:07:35.396888 (+ 322us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.397356 (+ 468us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:35.398144 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.398651 (+ 507us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:35.398659 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:35.398684 (+ 25us) server_negotiation.cc:300] Negotiation successful
0131 03:07:35.398721 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260131 03:07:35.398945 20521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8a3af424ce2461f945750680ea83063" candidate_uuid: "289c1997a4764445a603834638cbdf73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1af8dd1091db4983bffe39c4e3e03192" is_pre_election: true
I20260131 03:07:35.399086 20657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8a3af424ce2461f945750680ea83063" candidate_uuid: "289c1997a4764445a603834638cbdf73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b99e795d814af08c72129d249e5916" is_pre_election: true
I20260131 03:07:35.399120 20521 raft_consensus.cc:2468] T b8a3af424ce2461f945750680ea83063 P 1af8dd1091db4983bffe39c4e3e03192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289c1997a4764445a603834638cbdf73 in term 0.
I20260131 03:07:35.399205 20657 raft_consensus.cc:2468] T b8a3af424ce2461f945750680ea83063 P 18b99e795d814af08c72129d249e5916 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289c1997a4764445a603834638cbdf73 in term 0.
I20260131 03:07:35.399350 20726 leader_election.cc:304] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1af8dd1091db4983bffe39c4e3e03192, 289c1997a4764445a603834638cbdf73; no voters:
I20260131 03:07:35.399521 20864 raft_consensus.cc:2804] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:07:35.399596 20864 raft_consensus.cc:493] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:07:35.399632 20864 raft_consensus.cc:3060] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:35.400271 20864 raft_consensus.cc:515] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b99e795d814af08c72129d249e5916" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 41237 } } peers { permanent_uuid: "1af8dd1091db4983bffe39c4e3e03192" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 42601 } } peers { permanent_uuid: "289c1997a4764445a603834638cbdf73" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 40539 } }
I20260131 03:07:35.400439 20864 leader_election.cc:290] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73 [CANDIDATE]: Term 1 election: Requested vote from peers 18b99e795d814af08c72129d249e5916 (127.19.84.2:41237), 1af8dd1091db4983bffe39c4e3e03192 (127.19.84.1:42601)
I20260131 03:07:35.400614 20657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8a3af424ce2461f945750680ea83063" candidate_uuid: "289c1997a4764445a603834638cbdf73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b99e795d814af08c72129d249e5916"
I20260131 03:07:35.400724 20657 raft_consensus.cc:3060] T b8a3af424ce2461f945750680ea83063 P 18b99e795d814af08c72129d249e5916 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:35.400691 20521 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b8a3af424ce2461f945750680ea83063" candidate_uuid: "289c1997a4764445a603834638cbdf73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1af8dd1091db4983bffe39c4e3e03192"
I20260131 03:07:35.400799 20521 raft_consensus.cc:3060] T b8a3af424ce2461f945750680ea83063 P 1af8dd1091db4983bffe39c4e3e03192 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:35.401471 20657 raft_consensus.cc:2468] T b8a3af424ce2461f945750680ea83063 P 18b99e795d814af08c72129d249e5916 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289c1997a4764445a603834638cbdf73 in term 1.
I20260131 03:07:35.401605 20521 raft_consensus.cc:2468] T b8a3af424ce2461f945750680ea83063 P 1af8dd1091db4983bffe39c4e3e03192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289c1997a4764445a603834638cbdf73 in term 1.
I20260131 03:07:35.401708 20728 leader_election.cc:304] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18b99e795d814af08c72129d249e5916, 289c1997a4764445a603834638cbdf73; no voters:
I20260131 03:07:35.401805 20864 raft_consensus.cc:2804] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:07:35.401965 20864 raft_consensus.cc:697] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73 [term 1 LEADER]: Becoming Leader. State: Replica: 289c1997a4764445a603834638cbdf73, State: Running, Role: LEADER
I20260131 03:07:35.402225 20864 consensus_queue.cc:237] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b99e795d814af08c72129d249e5916" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 41237 } } peers { permanent_uuid: "1af8dd1091db4983bffe39c4e3e03192" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 42601 } } peers { permanent_uuid: "289c1997a4764445a603834638cbdf73" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 40539 } }
I20260131 03:07:35.403203 20374 catalog_manager.cc:5654] T b8a3af424ce2461f945750680ea83063 P 289c1997a4764445a603834638cbdf73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289c1997a4764445a603834638cbdf73 (127.19.84.3). New cstate: current_term: 1 leader_uuid: "289c1997a4764445a603834638cbdf73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b99e795d814af08c72129d249e5916" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1af8dd1091db4983bffe39c4e3e03192" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 42601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289c1997a4764445a603834638cbdf73" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 40539 } health_report { overall_health: HEALTHY } } }
Jan 31 03:07:35 dist-test-slave-5nnp krb5kdc[20326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.1@KRBTEST.COM
I20260131 03:07:35.418015 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:35.410528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56580 (local address 127.19.84.1:42601)
0131 03:07:35.410616 (+ 88us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:35.410619 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:35.410670 (+ 51us) server_negotiation.cc:408] Connection header received
0131 03:07:35.410760 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:35.410761 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:35.410770 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:35.410831 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:35.411582 (+ 751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.412043 (+ 461us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:35.412650 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.412856 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:35.415078 (+ 2222us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:35.415093 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:35.415094 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:35.415102 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:35.416949 (+ 1847us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:35.417306 (+ 357us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:35.417309 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:35.417310 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:35.417361 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:35.417573 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:35.417574 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:35.417575 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:35.417682 (+ 107us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:35.417823 (+ 141us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:35.417860 (+ 37us) server_negotiation.cc:300] Negotiation successful
0131 03:07:35.417941 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Jan 31 03:07:35 dist-test-slave-5nnp krb5kdc[20326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828855, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 31 03:07:35 dist-test-slave-5nnp krb5kdc[20326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828855, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.84.1@KRBTEST.COM
I20260131 03:07:35.440662 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:35.433329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56582 (local address 127.19.84.1:42601)
0131 03:07:35.433403 (+ 74us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:35.433405 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:35.433447 (+ 42us) server_negotiation.cc:408] Connection header received
0131 03:07:35.433519 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:35.433520 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:35.433558 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:35.433604 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:35.434351 (+ 747us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.434849 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:35.435583 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.435775 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:35.437905 (+ 2130us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:35.437909 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:35.437909 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:35.437916 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:35.439507 (+ 1591us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:35.439864 (+ 357us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:35.439865 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:35.439866 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:35.439892 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:35.440088 (+ 196us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:35.440089 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:35.440089 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:35.440165 (+ 76us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:35.440242 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:35.440431 (+ 189us) server_negotiation.cc:300] Negotiation successful
0131 03:07:35.440512 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Jan 31 03:07:35 dist-test-slave-5nnp krb5kdc[20326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 31 03:07:35 dist-test-slave-5nnp krb5kdc[20326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.1@KRBTEST.COM
I20260131 03:07:35.463526 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:35.456156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56592 (local address 127.19.84.1:42601)
0131 03:07:35.456234 (+ 78us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:35.456236 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:35.456268 (+ 32us) server_negotiation.cc:408] Connection header received
0131 03:07:35.456394 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:35.456396 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:35.456407 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:35.456466 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:35.457232 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.457696 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:35.458312 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:35.458448 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:35.460595 (+ 2147us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:35.460601 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:35.460601 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:35.460608 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:35.462311 (+ 1703us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:35.462678 (+ 367us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:35.462679 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:35.462680 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:35.462703 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:35.462886 (+ 183us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:35.462887 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:35.462887 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:35.462966 (+ 79us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:35.463041 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:35.463249 (+ 208us) server_negotiation.cc:300] Negotiation successful
0131 03:07:35.463362 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260131 03:07:35.464025 20481 tablet_service.cc:3012] Rejecting scan request for tablet b8a3af424ce2461f945750680ea83063: Uninitialized: clean time has not yet been initialized
I20260131 03:07:35.464956 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 20438
I20260131 03:07:35.469916 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 20574
I20260131 03:07:35.474565 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 20710
I20260131 03:07:35.479001 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 20342
2026-01-31T03:07:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3235 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:07:35 dist-test-slave-5nnp krb5kdc[20882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:07:35 dist-test-slave-5nnp krb5kdc[20882](info): set up 2 sockets
Jan 31 03:07:35 dist-test-slave-5nnp krb5kdc[20882](info): commencing operation
krb5kdc: starting...
W20260131 03:07:37.512394 19792 mini_kdc.cc:121] Time spent starting KDC: real 2.012s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:07:37 dist-test-slave-5nnp krb5kdc[20882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:07:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:07:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:37.650887 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:45499
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:33125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:45499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:37.732528 20898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:37.732796 20898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:37.732841 20898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:37.734352 20898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:07:37.734440 20898 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260131 03:07:37.734462 20898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:37.734484 20898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:37.734511 20898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:07:37.734540 20898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:07:37.736308 20898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:33125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:45499
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:45499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:37.736625 20898 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:37.736848 20898 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:37.739645 20903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:37.739642 20906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:37.739660 20904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:37.740166 20898 server_base.cc:1060] running on GCE node
I20260131 03:07:37.740406 20898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:37.740727 20898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:37.741986 20898 hybrid_clock.cc:648] HybridClock initialized: now 1769828857741971 us; error 27 us; skew 500 ppm
Jan 31 03:07:37 dist-test-slave-5nnp krb5kdc[20882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828857, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:37.744446 20898 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:07:37.745121 20898 webserver.cc:492] Webserver started at http://127.19.84.62:41397/ using document root <none> and password file <none>
I20260131 03:07:37.745318 20898 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:37.745373 20898 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:37.745505 20898 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:37.746552 20898 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "3f2e39897d584137a914e45c61e37146"
format_stamp: "Formatted at 2026-01-31 03:07:37 on dist-test-slave-5nnp"
server_key: "8e8304a50c90454e286290d2138fb05e"
server_key_iv: "49bf7509c428ebcaf606a5d46172a119"
server_key_version: "encryptionkey@0"
I20260131 03:07:37.746896 20898 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "3f2e39897d584137a914e45c61e37146"
format_stamp: "Formatted at 2026-01-31 03:07:37 on dist-test-slave-5nnp"
server_key: "8e8304a50c90454e286290d2138fb05e"
server_key_iv: "49bf7509c428ebcaf606a5d46172a119"
server_key_version: "encryptionkey@0"
I20260131 03:07:37.748297 20898 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:07:37.749188 20913 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:37.749437 20898 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:37.749567 20898 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "3f2e39897d584137a914e45c61e37146"
format_stamp: "Formatted at 2026-01-31 03:07:37 on dist-test-slave-5nnp"
server_key: "8e8304a50c90454e286290d2138fb05e"
server_key_iv: "49bf7509c428ebcaf606a5d46172a119"
server_key_version: "encryptionkey@0"
I20260131 03:07:37.749643 20898 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:37.758433 20898 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:37.760560 20898 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:37.760722 20898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:37.765856 20898 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:45499
I20260131 03:07:37.765901 20975 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:45499 every 8 connection(s)
I20260131 03:07:37.766227 20898 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:07:37.767083 20976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:37.769261 20976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146: Bootstrap starting.
I20260131 03:07:37.769863 20976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:37.770125 20976 log.cc:826] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146: Log is configured to *not* fsync() on all Append() calls
I20260131 03:07:37.770957 20976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146: No bootstrap required, opened a new log
I20260131 03:07:37.772236 20976 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f2e39897d584137a914e45c61e37146" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 45499 } }
I20260131 03:07:37.772356 20976 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:37.772377 20976 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f2e39897d584137a914e45c61e37146, State: Initialized, Role: FOLLOWER
I20260131 03:07:37.772528 20976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f2e39897d584137a914e45c61e37146" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 45499 } }
I20260131 03:07:37.772593 20976 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:07:37.772614 20976 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:07:37.772677 20976 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:37.773228 20976 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f2e39897d584137a914e45c61e37146" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 45499 } }
I20260131 03:07:37.773347 20976 leader_election.cc:304] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f2e39897d584137a914e45c61e37146; no voters:
I20260131 03:07:37.773519 20976 leader_election.cc:290] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:07:37.773640 20979 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:07:37.773779 20976 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:07:37.773811 20979 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146 [term 1 LEADER]: Becoming Leader. State: Replica: 3f2e39897d584137a914e45c61e37146, State: Running, Role: LEADER
I20260131 03:07:37.773908 20979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f2e39897d584137a914e45c61e37146" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 45499 } }
I20260131 03:07:37.774299 20980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f2e39897d584137a914e45c61e37146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f2e39897d584137a914e45c61e37146" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 45499 } } }
I20260131 03:07:37.774335 20981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f2e39897d584137a914e45c61e37146. Latest consensus state: current_term: 1 leader_uuid: "3f2e39897d584137a914e45c61e37146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f2e39897d584137a914e45c61e37146" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 45499 } } }
I20260131 03:07:37.774385 20980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146 [sys.catalog]: This master's current role is: LEADER
I20260131 03:07:37.774403 20981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146 [sys.catalog]: This master's current role is: LEADER
I20260131 03:07:37.774618 20987 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:07:37.775134 20987 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:07:37.775686 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 20898
I20260131 03:07:37.775774 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:07:37.775933 19792 external_mini_cluster.cc:1468] Setting key a4a92e8f26ba6f640248baf839a59a74
I20260131 03:07:37.779894 20987 catalog_manager.cc:1357] Generated new cluster ID: 7a891d8fd6e84a54a4830345a62258e5
I20260131 03:07:37.779985 20987 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Jan 31 03:07:37 dist-test-slave-5nnp krb5kdc[20882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:37.785866 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:37.776365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44686 (local address 127.19.84.62:45499)
0131 03:07:37.776513 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:37.776531 (+ 18us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:37.776552 (+ 21us) server_negotiation.cc:408] Connection header received
0131 03:07:37.776695 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:37.776700 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:37.776737 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:37.776843 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:37.777798 (+ 955us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:37.778540 (+ 742us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:37.779319 (+ 779us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:37.782041 (+ 2722us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:37.782159 (+ 118us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:37.782164 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:37.782165 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:37.782181 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:37.784301 (+ 2120us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:37.784718 (+ 417us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:37.784720 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:37.784721 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:37.784788 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:37.784965 (+ 177us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:37.784966 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:37.784967 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:37.785221 (+ 254us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:37.785352 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:37.785547 (+ 195us) server_negotiation.cc:300] Negotiation successful
0131 03:07:37.785680 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":64,"threads_started":1}
I20260131 03:07:37.790979 20987 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:07:37.791580 20987 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:07:37.805575 20987 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146: Generated new TSK 0
I20260131 03:07:37.805806 20987 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:37.855717 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:45499
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:33125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:37.938481 21004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:37.938663 21004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:37.938688 21004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:37.940310 21004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:37.940361 21004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:37.940382 21004 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260131 03:07:37.940402 21004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:07:37.942086 21004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:33125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.84.62:45499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:37.942343 21004 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:37.942562 21004 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:37.945360 21009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:37.945360 21012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:37.945461 21010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:37.945989 21004 server_base.cc:1060] running on GCE node
I20260131 03:07:37.946166 21004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:37.946360 21004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:37.947480 21004 hybrid_clock.cc:648] HybridClock initialized: now 1769828857947452 us; error 34 us; skew 500 ppm
Jan 31 03:07:37 dist-test-slave-5nnp krb5kdc[20882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828857, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:37.950228 21004 init.cc:377] Logged in from keytab as kudu/127.19.84.1@KRBTEST.COM (short username kudu)
I20260131 03:07:37.950917 21004 webserver.cc:492] Webserver started at http://127.19.84.1:43345/ using document root <none> and password file <none>
I20260131 03:07:37.951124 21004 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:37.951182 21004 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:37.951323 21004 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:37.952328 21004 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "30fbafdb859f4ccda011013c645c20c4"
format_stamp: "Formatted at 2026-01-31 03:07:37 on dist-test-slave-5nnp"
server_key: "273c001fc66bdcdf256cb4f9d74699bd"
server_key_iv: "cfb71b1d5041cb3d576e3576673007b8"
server_key_version: "encryptionkey@0"
I20260131 03:07:37.952670 21004 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "30fbafdb859f4ccda011013c645c20c4"
format_stamp: "Formatted at 2026-01-31 03:07:37 on dist-test-slave-5nnp"
server_key: "273c001fc66bdcdf256cb4f9d74699bd"
server_key_iv: "cfb71b1d5041cb3d576e3576673007b8"
server_key_version: "encryptionkey@0"
I20260131 03:07:37.954128 21004 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:07:37.955027 21019 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:37.955257 21004 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:37.955354 21004 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "30fbafdb859f4ccda011013c645c20c4"
format_stamp: "Formatted at 2026-01-31 03:07:37 on dist-test-slave-5nnp"
server_key: "273c001fc66bdcdf256cb4f9d74699bd"
server_key_iv: "cfb71b1d5041cb3d576e3576673007b8"
server_key_version: "encryptionkey@0"
I20260131 03:07:37.955613 21004 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:37.968842 21004 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:37.971182 21004 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:37.971361 21004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:37.972049 21004 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:07:37.972107 21004 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:37.972150 21004 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:07:37.972173 21004 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 31 03:07:37 dist-test-slave-5nnp krb5kdc[20882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828857, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:37.979888 21004 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:39417
I20260131 03:07:37.979972 21134 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:39417 every 8 connection(s)
I20260131 03:07:37.980263 21004 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
I20260131 03:07:37.980561 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 21004
I20260131 03:07:37.980644 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:07:37.980800 19792 external_mini_cluster.cc:1468] Setting key 0d162a35ec41f6f50f469ed3fd6cb397
I20260131 03:07:37.981571 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:37.972543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:49607 (local address 127.19.84.62:45499)
0131 03:07:37.972643 (+ 100us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:37.972645 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:37.972856 (+ 211us) server_negotiation.cc:408] Connection header received
0131 03:07:37.973565 (+ 709us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:37.973567 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:37.973581 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:37.973682 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:37.974793 (+ 1111us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:37.975585 (+ 792us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:37.976245 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:37.976413 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:37.978342 (+ 1929us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:37.978347 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:37.978347 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:37.978355 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:37.979934 (+ 1579us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:37.980336 (+ 402us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:37.980337 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:37.980338 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:37.980372 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:37.980651 (+ 279us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:37.980652 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:37.980653 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:37.980879 (+ 226us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:37.980999 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:37.981359 (+ 360us) server_negotiation.cc:300] Negotiation successful
0131 03:07:37.981474 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260131 03:07:37.981665 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:37.972518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:45499 (local address 127.19.84.1:49607)
0131 03:07:37.972698 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:37.972741 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:37.973459 (+ 718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:37.973746 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:37.973754 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:37.974238 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:37.974683 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:37.974689 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:37.975688 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:37.975689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:37.976144 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:37.976146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:37.976317 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:37.976878 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:37.976897 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:37.978247 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:37.980037 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:37.980042 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:37.980045 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:37.980252 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:37.980470 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:37.980470 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:37.980471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:37.980566 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:37.980987 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:37.980993 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:37.981207 (+ 214us) client_negotiation.cc:769] Sending connection context
0131 03:07:37.981325 (+ 118us) client_negotiation.cc:240] Negotiation successful
0131 03:07:37.981474 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260131 03:07:37.982187 21135 heartbeater.cc:344] Connected to a master server at 127.19.84.62:45499
I20260131 03:07:37.982306 21135 heartbeater.cc:461] Registering TS with master...
I20260131 03:07:37.982525 21135 heartbeater.cc:507] Master 127.19.84.62:45499 requested a full tablet report, sending...
I20260131 03:07:37.983906 20930 ts_manager.cc:194] Registered new tserver with Master: 30fbafdb859f4ccda011013c645c20c4 (127.19.84.1:39417)
I20260131 03:07:37.984558 20930 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.1@KRBTEST.COM'} at 127.19.84.1:49607
WARNING: no policy specified for kudu/127.19.84.2@KRBTEST.COM; defaulting to no policy
I20260131 03:07:37.990079 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:37.984705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:56945 (local address 127.19.84.62:45499)
0131 03:07:37.984804 (+ 99us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:37.984807 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:37.984816 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:07:37.984847 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:37.984848 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:37.984857 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:37.984916 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:37.985398 (+ 482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:37.985872 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:37.986442 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:37.986648 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:37.987495 (+ 847us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:37.987502 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:37.987502 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:37.987513 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:37.988944 (+ 1431us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:37.989376 (+ 432us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:37.989378 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:37.989378 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:37.989411 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:37.989570 (+ 159us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:37.989570 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:37.989571 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:37.989674 (+ 103us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:37.989747 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:37.989895 (+ 148us) server_negotiation.cc:300] Negotiation successful
0131 03:07:37.989978 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260131 03:07:37.990094 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:37.983827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:45499 (local address 127.19.84.1:56945)
0131 03:07:37.984001 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:37.984013 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:37.984075 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:37.984977 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:37.984978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:37.985133 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:37.985304 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:37.985306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:37.985965 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:37.985966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:37.986352 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:37.986354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:37.986446 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:37.987062 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:37.987066 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:37.987329 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:37.989073 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:37.989074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:37.989075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:37.989286 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:37.989479 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:37.989480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:37.989480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:37.989514 (+ 34us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:37.989736 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:37.989737 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:37.989783 (+ 46us) client_negotiation.cc:769] Sending connection context
0131 03:07:37.989852 (+ 69us) client_negotiation.cc:240] Negotiation successful
0131 03:07:37.989973 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:38.032390 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:45499
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:33125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:38.112481 21146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:38.112709 21146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:38.112749 21146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:38.114219 21146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:38.114292 21146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:38.114328 21146 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260131 03:07:38.114368 21146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:07:38.116003 21146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:33125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.84.62:45499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:38.116267 21146 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:38.116466 21146 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:38.119427 21152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:38.119443 21151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:38.119445 21154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:38.119711 21146 server_base.cc:1060] running on GCE node
I20260131 03:07:38.119874 21146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:38.120057 21146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:38.121188 21146 hybrid_clock.cc:648] HybridClock initialized: now 1769828858121157 us; error 47 us; skew 500 ppm
Jan 31 03:07:38 dist-test-slave-5nnp krb5kdc[20882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828858, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:38.123725 21146 init.cc:377] Logged in from keytab as kudu/127.19.84.2@KRBTEST.COM (short username kudu)
I20260131 03:07:38.124270 21146 webserver.cc:492] Webserver started at http://127.19.84.2:37215/ using document root <none> and password file <none>
I20260131 03:07:38.124480 21146 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:38.124533 21146 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:38.124704 21146 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:38.125677 21146 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "0e3bd778cdc54908bfd908adb74b0d0b"
format_stamp: "Formatted at 2026-01-31 03:07:38 on dist-test-slave-5nnp"
server_key: "fc8239e24b7ab5d3f6bd5ed070f7aec4"
server_key_iv: "45fed66981503edec4710f753d2659a8"
server_key_version: "encryptionkey@0"
I20260131 03:07:38.126010 21146 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "0e3bd778cdc54908bfd908adb74b0d0b"
format_stamp: "Formatted at 2026-01-31 03:07:38 on dist-test-slave-5nnp"
server_key: "fc8239e24b7ab5d3f6bd5ed070f7aec4"
server_key_iv: "45fed66981503edec4710f753d2659a8"
server_key_version: "encryptionkey@0"
I20260131 03:07:38.127226 21146 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260131 03:07:38.128100 21161 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:38.128299 21146 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:38.128381 21146 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "0e3bd778cdc54908bfd908adb74b0d0b"
format_stamp: "Formatted at 2026-01-31 03:07:38 on dist-test-slave-5nnp"
server_key: "fc8239e24b7ab5d3f6bd5ed070f7aec4"
server_key_iv: "45fed66981503edec4710f753d2659a8"
server_key_version: "encryptionkey@0"
I20260131 03:07:38.128464 21146 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:38.150861 21146 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:38.152899 21146 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:38.153059 21146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:38.153621 21146 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:07:38.153678 21146 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:38.153723 21146 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:07:38.153749 21146 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 31 03:07:38 dist-test-slave-5nnp krb5kdc[20882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828858, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:38.161415 21146 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:45829
I20260131 03:07:38.161471 21276 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:45829 every 8 connection(s)
I20260131 03:07:38.161841 21146 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
I20260131 03:07:38.163525 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.154271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:56129 (local address 127.19.84.62:45499)
0131 03:07:38.154350 (+ 79us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:38.154352 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:38.154475 (+ 123us) server_negotiation.cc:408] Connection header received
0131 03:07:38.154992 (+ 517us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:38.154993 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:38.155007 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:38.155092 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:38.155983 (+ 891us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.156839 (+ 856us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.157717 (+ 878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.157899 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.160640 (+ 2741us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:38.160645 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:38.160646 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:38.160655 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:38.162420 (+ 1765us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:38.162778 (+ 358us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:38.162779 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:38.162779 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:38.162802 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:38.162972 (+ 170us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:38.162972 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:38.162972 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:38.163061 (+ 89us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:38.163111 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:38.163346 (+ 235us) server_negotiation.cc:300] Negotiation successful
0131 03:07:38.163460 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260131 03:07:38.163658 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.154194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:45499 (local address 127.19.84.2:56129)
0131 03:07:38.154358 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:38.154389 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:38.154875 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:38.155161 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:38.155164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:38.155581 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:38.155901 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.155904 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.156940 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.156941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:38.157589 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.157591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.157784 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.158371 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:38.158398 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:38.160524 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:38.162496 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:38.162499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:38.162502 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:38.162699 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:38.162866 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:38.162866 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:38.162867 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:38.162920 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:38.163113 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:38.163116 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:38.163236 (+ 120us) client_negotiation.cc:769] Sending connection context
0131 03:07:38.163323 (+ 87us) client_negotiation.cc:240] Negotiation successful
0131 03:07:38.163495 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":107,"threads_started":1}
I20260131 03:07:38.164973 21277 heartbeater.cc:344] Connected to a master server at 127.19.84.62:45499
I20260131 03:07:38.165076 21277 heartbeater.cc:461] Registering TS with master...
I20260131 03:07:38.165284 21277 heartbeater.cc:507] Master 127.19.84.62:45499 requested a full tablet report, sending...
I20260131 03:07:38.165663 20929 ts_manager.cc:194] Registered new tserver with Master: 0e3bd778cdc54908bfd908adb74b0d0b (127.19.84.2:45829)
I20260131 03:07:38.166095 20929 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.2@KRBTEST.COM'} at 127.19.84.2:56129
I20260131 03:07:38.168640 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 21146
I20260131 03:07:38.168764 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:07:38.168948 19792 external_mini_cluster.cc:1468] Setting key d6a813c861509ff9dc9774fa5add84ee
I20260131 03:07:38.171133 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.165036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:34807 (local address 127.19.84.62:45499)
0131 03:07:38.165080 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:38.165082 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:38.165689 (+ 607us) server_negotiation.cc:408] Connection header received
0131 03:07:38.165780 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:38.165781 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:38.165791 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:38.165847 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:38.166335 (+ 488us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.166875 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.167574 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.167761 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.168428 (+ 667us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:38.168432 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:38.168432 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:38.168438 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:38.169963 (+ 1525us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:38.170343 (+ 380us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:38.170344 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:38.170344 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:38.170366 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:38.170581 (+ 215us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:38.170582 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:38.170582 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:38.170691 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:38.170750 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:38.170975 (+ 225us) server_negotiation.cc:300] Negotiation successful
0131 03:07:38.171053 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260131 03:07:38.171463 21283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.164977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:45499 (local address 127.19.84.2:34807)
0131 03:07:38.165502 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:38.165514 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:38.165712 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:38.165848 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:38.165849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:38.166004 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:38.166173 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.166175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.167017 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.167018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:38.167457 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.167459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.167525 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.168083 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:38.168088 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:38.168333 (+ 245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:38.170058 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:38.170059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:38.170060 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:38.170263 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:38.170444 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:38.170445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:38.170445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:38.170494 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:38.170758 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:38.170759 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:38.170835 (+ 76us) client_negotiation.cc:769] Sending connection context
0131 03:07:38.170909 (+ 74us) client_negotiation.cc:240] Negotiation successful
0131 03:07:38.171027 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:38.218796 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:45499
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:33125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:38.299577 21288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:38.299760 21288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:38.299785 21288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:38.301198 21288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:38.301244 21288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:38.301265 21288 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260131 03:07:38.301282 21288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:07:38.302958 21288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:33125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.84.62:45499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:38.303207 21288 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:38.303448 21288 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260131 03:07:38.306193 21288 server_base.cc:1060] running on GCE node
W20260131 03:07:38.306198 21294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:38.306272 21293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:38.306304 21296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:38.306569 21288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:38.306836 21288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:38.307998 21288 hybrid_clock.cc:648] HybridClock initialized: now 1769828858307976 us; error 37 us; skew 500 ppm
Jan 31 03:07:38 dist-test-slave-5nnp krb5kdc[20882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828858, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:38.310528 21288 init.cc:377] Logged in from keytab as kudu/127.19.84.3@KRBTEST.COM (short username kudu)
I20260131 03:07:38.311177 21288 webserver.cc:492] Webserver started at http://127.19.84.3:41199/ using document root <none> and password file <none>
I20260131 03:07:38.311444 21288 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:38.311527 21288 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:38.311656 21288 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:38.312630 21288 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "9de879312e9b477d977409365f1425d1"
format_stamp: "Formatted at 2026-01-31 03:07:38 on dist-test-slave-5nnp"
server_key: "02e05faef61feb402012b064078074ee"
server_key_iv: "564f3434bcb2d5ea3e27b63b12f78f7d"
server_key_version: "encryptionkey@0"
I20260131 03:07:38.312978 21288 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "9de879312e9b477d977409365f1425d1"
format_stamp: "Formatted at 2026-01-31 03:07:38 on dist-test-slave-5nnp"
server_key: "02e05faef61feb402012b064078074ee"
server_key_iv: "564f3434bcb2d5ea3e27b63b12f78f7d"
server_key_version: "encryptionkey@0"
I20260131 03:07:38.314213 21288 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:07:38.314921 21303 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:38.315104 21288 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260131 03:07:38.315196 21288 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "9de879312e9b477d977409365f1425d1"
format_stamp: "Formatted at 2026-01-31 03:07:38 on dist-test-slave-5nnp"
server_key: "02e05faef61feb402012b064078074ee"
server_key_iv: "564f3434bcb2d5ea3e27b63b12f78f7d"
server_key_version: "encryptionkey@0"
I20260131 03:07:38.315276 21288 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:38.337837 21288 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:38.340106 21288 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:38.340284 21288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:38.340935 21288 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:07:38.341013 21288 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:38.341059 21288 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:07:38.341101 21288 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 31 03:07:38 dist-test-slave-5nnp krb5kdc[20882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828858, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:38.348671 21288 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:36581
I20260131 03:07:38.348758 21418 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:36581 every 8 connection(s)
I20260131 03:07:38.349072 21288 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
I20260131 03:07:38.350417 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.341298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:40511 (local address 127.19.84.62:45499)
0131 03:07:38.341383 (+ 85us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:38.341386 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:38.341602 (+ 216us) server_negotiation.cc:408] Connection header received
0131 03:07:38.342401 (+ 799us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:38.342402 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:38.342416 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:38.342490 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:38.343717 (+ 1227us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.344329 (+ 612us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.344959 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.345121 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.347130 (+ 2009us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:38.347135 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:38.347136 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:38.347144 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:38.349166 (+ 2022us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:38.349673 (+ 507us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:38.349675 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:38.349675 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:38.349701 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:38.349870 (+ 169us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:38.349871 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:38.349871 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:38.349961 (+ 90us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:38.350005 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:38.350243 (+ 238us) server_negotiation.cc:300] Negotiation successful
0131 03:07:38.350353 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260131 03:07:38.350474 21312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.341276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:45499 (local address 127.19.84.3:40511)
0131 03:07:38.341453 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:38.341490 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:38.342237 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:38.342581 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:38.342586 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:38.343096 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:38.343585 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.343591 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.344437 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.344439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:38.344861 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.344863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.345025 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.345564 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:38.345590 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:38.347008 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:38.349266 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:38.349271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:38.349274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:38.349567 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:38.349763 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:38.349763 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:38.349764 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:38.349822 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:38.350008 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:38.350011 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:38.350131 (+ 120us) client_negotiation.cc:769] Sending connection context
0131 03:07:38.350215 (+ 84us) client_negotiation.cc:240] Negotiation successful
0131 03:07:38.350320 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
I20260131 03:07:38.351063 21419 heartbeater.cc:344] Connected to a master server at 127.19.84.62:45499
I20260131 03:07:38.351155 21419 heartbeater.cc:461] Registering TS with master...
I20260131 03:07:38.351366 21419 heartbeater.cc:507] Master 127.19.84.62:45499 requested a full tablet report, sending...
I20260131 03:07:38.352015 20929 ts_manager.cc:194] Registered new tserver with Master: 9de879312e9b477d977409365f1425d1 (127.19.84.3:36581)
I20260131 03:07:38.352628 20929 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:40511
I20260131 03:07:38.353681 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 21288
I20260131 03:07:38.353768 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:07:38.353917 19792 external_mini_cluster.cc:1468] Setting key 28ca7584dc35c16a0a389a4e2daa5ec4
I20260131 03:07:38.354823 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:07:38.358323 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.352319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:37989 (local address 127.19.84.62:45499)
0131 03:07:38.352374 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:38.352376 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:38.352544 (+ 168us) server_negotiation.cc:408] Connection header received
0131 03:07:38.352569 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:38.352570 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:38.352581 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:38.352643 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:38.353107 (+ 464us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.353618 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.354317 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.354880 (+ 563us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.355299 (+ 419us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:38.355304 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:38.355304 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:38.355311 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:38.357121 (+ 1810us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:38.357522 (+ 401us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:38.357523 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:38.357523 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:38.357551 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:38.357726 (+ 175us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:38.357728 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:38.357728 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:38.357844 (+ 116us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:38.357897 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:38.358100 (+ 203us) server_negotiation.cc:300] Negotiation successful
0131 03:07:38.358191 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260131 03:07:38.358662 21425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.352253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:45499 (local address 127.19.84.3:37989)
0131 03:07:38.352431 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:38.352441 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:38.352496 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:38.352645 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:38.352645 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:38.352829 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:38.353020 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.353022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.353702 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.353703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:38.354220 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.354222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.354295 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.354828 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:38.354832 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:38.355171 (+ 339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:38.357223 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:38.357225 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:38.357225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:38.357425 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:38.357612 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:38.357614 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:38.357614 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:38.357653 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:38.357943 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:38.357944 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:38.357994 (+ 50us) client_negotiation.cc:769] Sending connection context
0131 03:07:38.358427 (+ 433us) client_negotiation.cc:240] Negotiation successful
0131 03:07:38.358551 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":77,"threads_started":1}
Jan 31 03:07:38 dist-test-slave-5nnp krb5kdc[20882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828858, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 31 03:07:38 dist-test-slave-5nnp krb5kdc[20882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828858, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:38.379118 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.371933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44688 (local address 127.19.84.62:45499)
0131 03:07:38.372006 (+ 73us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:38.372008 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:38.372017 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:07:38.372043 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:38.372044 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:38.372052 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:38.372115 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:38.372912 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.373402 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.374015 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.374211 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.376636 (+ 2425us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:38.376641 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:38.376641 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:38.376649 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:38.378104 (+ 1455us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:38.378443 (+ 339us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:38.378445 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:38.378445 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:38.378469 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:38.378661 (+ 192us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:38.378662 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:38.378662 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:38.378758 (+ 96us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:38.378843 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:38.378965 (+ 122us) server_negotiation.cc:300] Negotiation successful
0131 03:07:38.379035 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260131 03:07:38.381174 20929 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:07:38.382516 20929 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:07:38.388222 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.384970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:36581 (local address 127.0.0.1:50558)
0131 03:07:38.385273 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:38.385282 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:38.385357 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:38.385680 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:38.385681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:38.385683 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:07:38.385880 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.385882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.387163 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.387164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:38.388046 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.388048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.388107 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.388109 (+ 2us) client_negotiation.cc:769] Sending connection context
0131 03:07:38.388138 (+ 29us) client_negotiation.cc:240] Negotiation successful
0131 03:07:38.388169 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":77,"threads_started":1}
I20260131 03:07:38.388221 21437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.384941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:45829 (local address 127.0.0.1:57564)
0131 03:07:38.385206 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:38.385240 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:38.385328 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:38.385556 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:38.385560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:38.385563 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:07:38.385864 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.385867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.386971 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.386972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:38.387906 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.387908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.387984 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.388023 (+ 39us) client_negotiation.cc:769] Sending connection context
0131 03:07:38.388074 (+ 51us) client_negotiation.cc:240] Negotiation successful
0131 03:07:38.388118 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":71,"threads_started":1}
I20260131 03:07:38.388983 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.385360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52800 (local address 127.19.84.1:39417)
0131 03:07:38.385553 (+ 193us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:38.385561 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:38.385573 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:07:38.385639 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:38.385641 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:38.385657 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:38.385722 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:07:38.386152 (+ 430us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.387252 (+ 1100us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.388227 (+ 975us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.388767 (+ 540us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.388823 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:38.388858 (+ 35us) server_negotiation.cc:300] Negotiation successful
0131 03:07:38.388902 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":96,"threads_started":1}
I20260131 03:07:38.388990 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.385017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.19.84.2:45829)
0131 03:07:38.385206 (+ 189us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:38.385213 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:38.385323 (+ 110us) server_negotiation.cc:408] Connection header received
0131 03:07:38.385404 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:38.385407 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:38.385426 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:38.385939 (+ 513us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:07:38.386035 (+ 96us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.386881 (+ 846us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.388332 (+ 1451us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.388820 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.388854 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:38.388880 (+ 26us) server_negotiation.cc:300] Negotiation successful
0131 03:07:38.388917 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
I20260131 03:07:38.389137 21439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.385052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50558 (local address 127.19.84.3:36581)
0131 03:07:38.385468 (+ 416us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:38.385476 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:38.385485 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:07:38.385526 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:38.385529 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:38.385605 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:38.385679 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:07:38.385962 (+ 283us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.387083 (+ 1121us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.388410 (+ 1327us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.388934 (+ 524us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.388967 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:38.388991 (+ 24us) server_negotiation.cc:300] Negotiation successful
0131 03:07:38.389018 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":103,"threads_started":1}
I20260131 03:07:38.389349 21436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.385160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:39417 (local address 127.0.0.1:52800)
0131 03:07:38.385515 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:38.385525 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:38.385574 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:38.385822 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:38.385822 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:38.385824 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:07:38.386000 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.386001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.387338 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.387339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:38.388008 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.388011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.388080 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.388084 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:07:38.388119 (+ 35us) client_negotiation.cc:240] Negotiation successful
0131 03:07:38.388166 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":84,"threads_started":1}
I20260131 03:07:38.389613 21211 tablet_service.cc:1505] Processing CreateTablet for tablet af7b5d27345443d4ba1889f61e3a1ec0 (DEFAULT_TABLE table=test-table [id=9441163cdf6f41b1975ace6d263bf7b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:07:38.389642 21353 tablet_service.cc:1505] Processing CreateTablet for tablet af7b5d27345443d4ba1889f61e3a1ec0 (DEFAULT_TABLE table=test-table [id=9441163cdf6f41b1975ace6d263bf7b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:07:38.389916 21211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af7b5d27345443d4ba1889f61e3a1ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:38.389925 21353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af7b5d27345443d4ba1889f61e3a1ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:38.392334 21443 tablet_bootstrap.cc:492] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b: Bootstrap starting.
I20260131 03:07:38.392664 21069 tablet_service.cc:1505] Processing CreateTablet for tablet af7b5d27345443d4ba1889f61e3a1ec0 (DEFAULT_TABLE table=test-table [id=9441163cdf6f41b1975ace6d263bf7b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:07:38.392925 21069 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af7b5d27345443d4ba1889f61e3a1ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:38.392959 21443 tablet_bootstrap.cc:654] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:38.393518 21443 log.cc:826] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b: Log is configured to *not* fsync() on all Append() calls
I20260131 03:07:38.393828 21442 tablet_bootstrap.cc:492] T af7b5d27345443d4ba1889f61e3a1ec0 P 9de879312e9b477d977409365f1425d1: Bootstrap starting.
I20260131 03:07:38.394347 21443 tablet_bootstrap.cc:492] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b: No bootstrap required, opened a new log
I20260131 03:07:38.394487 21443 ts_tablet_manager.cc:1403] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:07:38.394446 21442 tablet_bootstrap.cc:654] T af7b5d27345443d4ba1889f61e3a1ec0 P 9de879312e9b477d977409365f1425d1: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:38.395195 21442 log.cc:826] T af7b5d27345443d4ba1889f61e3a1ec0 P 9de879312e9b477d977409365f1425d1: Log is configured to *not* fsync() on all Append() calls
I20260131 03:07:38.395282 21445 tablet_bootstrap.cc:492] T af7b5d27345443d4ba1889f61e3a1ec0 P 30fbafdb859f4ccda011013c645c20c4: Bootstrap starting.
I20260131 03:07:38.395895 21445 tablet_bootstrap.cc:654] T af7b5d27345443d4ba1889f61e3a1ec0 P 30fbafdb859f4ccda011013c645c20c4: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:38.395995 21442 tablet_bootstrap.cc:492] T af7b5d27345443d4ba1889f61e3a1ec0 P 9de879312e9b477d977409365f1425d1: No bootstrap required, opened a new log
I20260131 03:07:38.396068 21442 ts_tablet_manager.cc:1403] T af7b5d27345443d4ba1889f61e3a1ec0 P 9de879312e9b477d977409365f1425d1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260131 03:07:38.396425 21443 raft_consensus.cc:359] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.396541 21445 log.cc:826] T af7b5d27345443d4ba1889f61e3a1ec0 P 30fbafdb859f4ccda011013c645c20c4: Log is configured to *not* fsync() on all Append() calls
I20260131 03:07:38.396580 21443 raft_consensus.cc:385] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:38.396620 21443 raft_consensus.cc:740] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e3bd778cdc54908bfd908adb74b0d0b, State: Initialized, Role: FOLLOWER
I20260131 03:07:38.396730 21443 consensus_queue.cc:260] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.396982 21443 ts_tablet_manager.cc:1434] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:07:38.397104 21277 heartbeater.cc:499] Master 127.19.84.62:45499 was elected leader, sending a full tablet report...
I20260131 03:07:38.397374 21445 tablet_bootstrap.cc:492] T af7b5d27345443d4ba1889f61e3a1ec0 P 30fbafdb859f4ccda011013c645c20c4: No bootstrap required, opened a new log
I20260131 03:07:38.397439 21445 ts_tablet_manager.cc:1403] T af7b5d27345443d4ba1889f61e3a1ec0 P 30fbafdb859f4ccda011013c645c20c4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:07:38.397712 21442 raft_consensus.cc:359] T af7b5d27345443d4ba1889f61e3a1ec0 P 9de879312e9b477d977409365f1425d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.397859 21442 raft_consensus.cc:385] T af7b5d27345443d4ba1889f61e3a1ec0 P 9de879312e9b477d977409365f1425d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:38.397905 21442 raft_consensus.cc:740] T af7b5d27345443d4ba1889f61e3a1ec0 P 9de879312e9b477d977409365f1425d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9de879312e9b477d977409365f1425d1, State: Initialized, Role: FOLLOWER
I20260131 03:07:38.398000 21442 consensus_queue.cc:260] T af7b5d27345443d4ba1889f61e3a1ec0 P 9de879312e9b477d977409365f1425d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.398247 21442 ts_tablet_manager.cc:1434] T af7b5d27345443d4ba1889f61e3a1ec0 P 9de879312e9b477d977409365f1425d1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260131 03:07:38.398298 21419 heartbeater.cc:499] Master 127.19.84.62:45499 was elected leader, sending a full tablet report...
I20260131 03:07:38.399461 21445 raft_consensus.cc:359] T af7b5d27345443d4ba1889f61e3a1ec0 P 30fbafdb859f4ccda011013c645c20c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.399597 21445 raft_consensus.cc:385] T af7b5d27345443d4ba1889f61e3a1ec0 P 30fbafdb859f4ccda011013c645c20c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:38.399637 21445 raft_consensus.cc:740] T af7b5d27345443d4ba1889f61e3a1ec0 P 30fbafdb859f4ccda011013c645c20c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30fbafdb859f4ccda011013c645c20c4, State: Initialized, Role: FOLLOWER
I20260131 03:07:38.399739 21445 consensus_queue.cc:260] T af7b5d27345443d4ba1889f61e3a1ec0 P 30fbafdb859f4ccda011013c645c20c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.399972 21445 ts_tablet_manager.cc:1434] T af7b5d27345443d4ba1889f61e3a1ec0 P 30fbafdb859f4ccda011013c645c20c4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:07:38.400053 21135 heartbeater.cc:499] Master 127.19.84.62:45499 was elected leader, sending a full tablet report...
W20260131 03:07:38.412366 21278 tablet.cc:2378] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260131 03:07:38.481546 21136 tablet.cc:2378] T af7b5d27345443d4ba1889f61e3a1ec0 P 30fbafdb859f4ccda011013c645c20c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260131 03:07:38.599726 21420 tablet.cc:2378] T af7b5d27345443d4ba1889f61e3a1ec0 P 9de879312e9b477d977409365f1425d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:07:38.692103 21448 raft_consensus.cc:493] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:07:38.692241 21448 raft_consensus.cc:515] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.692636 21448 leader_election.cc:290] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30fbafdb859f4ccda011013c645c20c4 (127.19.84.1:39417), 9de879312e9b477d977409365f1425d1 (127.19.84.3:36581)
I20260131 03:07:38.694988 21452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.692924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:39417 (local address 127.19.84.2:57705)
0131 03:07:38.693049 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:38.693060 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:38.693151 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:38.693308 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:38.693309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:38.693311 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:07:38.693519 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.693521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.694153 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.694154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:38.694714 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.694716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.694810 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.694822 (+ 12us) client_negotiation.cc:769] Sending connection context
0131 03:07:38.694855 (+ 33us) client_negotiation.cc:240] Negotiation successful
0131 03:07:38.694896 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260131 03:07:38.695200 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.692843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:36581 (local address 127.19.84.2:46829)
0131 03:07:38.692964 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:38.692977 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:38.693045 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:38.693208 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:38.693209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:38.693212 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:07:38.693386 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.693388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.694073 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.694074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:38.694789 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.694792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.694933 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.694939 (+ 6us) client_negotiation.cc:769] Sending connection context
0131 03:07:38.694974 (+ 35us) client_negotiation.cc:240] Negotiation successful
0131 03:07:38.695013 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
I20260131 03:07:38.695469 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.692975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:57705 (local address 127.19.84.1:39417)
0131 03:07:38.693047 (+ 72us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:38.693049 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:38.693133 (+ 84us) server_negotiation.cc:408] Connection header received
0131 03:07:38.693219 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:38.693221 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:38.693230 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:38.693304 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:07:38.693604 (+ 300us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.694066 (+ 462us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.694808 (+ 742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.695254 (+ 446us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.695264 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:38.695289 (+ 25us) server_negotiation.cc:300] Negotiation successful
0131 03:07:38.695334 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260131 03:07:38.695665 21439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.692981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:46829 (local address 127.19.84.3:36581)
0131 03:07:38.693052 (+ 71us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:38.693054 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:38.693082 (+ 28us) server_negotiation.cc:408] Connection header received
0131 03:07:38.693112 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:38.693113 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:38.693124 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:38.693199 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:07:38.693471 (+ 272us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.693979 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.694952 (+ 973us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.695487 (+ 535us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.695494 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:38.695556 (+ 62us) server_negotiation.cc:300] Negotiation successful
0131 03:07:38.695606 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260131 03:07:38.695782 21089 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af7b5d27345443d4ba1889f61e3a1ec0" candidate_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30fbafdb859f4ccda011013c645c20c4" is_pre_election: true
I20260131 03:07:38.695979 21089 raft_consensus.cc:2468] T af7b5d27345443d4ba1889f61e3a1ec0 P 30fbafdb859f4ccda011013c645c20c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e3bd778cdc54908bfd908adb74b0d0b in term 0.
I20260131 03:07:38.695998 21373 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af7b5d27345443d4ba1889f61e3a1ec0" candidate_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9de879312e9b477d977409365f1425d1" is_pre_election: true
I20260131 03:07:38.696161 21373 raft_consensus.cc:2468] T af7b5d27345443d4ba1889f61e3a1ec0 P 9de879312e9b477d977409365f1425d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e3bd778cdc54908bfd908adb74b0d0b in term 0.
I20260131 03:07:38.696233 21162 leader_election.cc:304] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e3bd778cdc54908bfd908adb74b0d0b, 30fbafdb859f4ccda011013c645c20c4; no voters:
I20260131 03:07:38.696382 21448 raft_consensus.cc:2804] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:07:38.696471 21448 raft_consensus.cc:493] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:07:38.696523 21448 raft_consensus.cc:3060] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:38.697158 21448 raft_consensus.cc:515] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.697340 21448 leader_election.cc:290] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b [CANDIDATE]: Term 1 election: Requested vote from peers 30fbafdb859f4ccda011013c645c20c4 (127.19.84.1:39417), 9de879312e9b477d977409365f1425d1 (127.19.84.3:36581)
I20260131 03:07:38.697504 21373 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af7b5d27345443d4ba1889f61e3a1ec0" candidate_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9de879312e9b477d977409365f1425d1"
I20260131 03:07:38.697541 21089 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af7b5d27345443d4ba1889f61e3a1ec0" candidate_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30fbafdb859f4ccda011013c645c20c4"
I20260131 03:07:38.697628 21373 raft_consensus.cc:3060] T af7b5d27345443d4ba1889f61e3a1ec0 P 9de879312e9b477d977409365f1425d1 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:38.697626 21089 raft_consensus.cc:3060] T af7b5d27345443d4ba1889f61e3a1ec0 P 30fbafdb859f4ccda011013c645c20c4 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:38.698372 21373 raft_consensus.cc:2468] T af7b5d27345443d4ba1889f61e3a1ec0 P 9de879312e9b477d977409365f1425d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e3bd778cdc54908bfd908adb74b0d0b in term 1.
I20260131 03:07:38.698396 21089 raft_consensus.cc:2468] T af7b5d27345443d4ba1889f61e3a1ec0 P 30fbafdb859f4ccda011013c645c20c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e3bd778cdc54908bfd908adb74b0d0b in term 1.
I20260131 03:07:38.698652 21162 leader_election.cc:304] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e3bd778cdc54908bfd908adb74b0d0b, 30fbafdb859f4ccda011013c645c20c4; no voters:
I20260131 03:07:38.698890 21448 raft_consensus.cc:2804] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:07:38.699046 21448 raft_consensus.cc:697] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b [term 1 LEADER]: Becoming Leader. State: Replica: 0e3bd778cdc54908bfd908adb74b0d0b, State: Running, Role: LEADER
I20260131 03:07:38.699158 21448 consensus_queue.cc:237] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.700137 20929 catalog_manager.cc:5654] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e3bd778cdc54908bfd908adb74b0d0b (127.19.84.2). New cstate: current_term: 1 leader_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } health_report { overall_health: UNKNOWN } } }
I20260131 03:07:38.744732 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.742437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57574 (local address 127.19.84.2:45829)
0131 03:07:38.742502 (+ 65us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:38.742504 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:38.742514 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:07:38.742596 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:38.742597 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:38.742615 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:38.742668 (+ 53us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:07:38.742959 (+ 291us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.743555 (+ 596us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.744176 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.744330 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.744364 (+ 34us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:07:38.744453 (+ 89us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:07:38.744561 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:38.744605 (+ 44us) server_negotiation.cc:300] Negotiation successful
0131 03:07:38.744639 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260131 03:07:38.746459 21373 raft_consensus.cc:1275] T af7b5d27345443d4ba1889f61e3a1ec0 P 9de879312e9b477d977409365f1425d1 [term 1 FOLLOWER]: Refusing update from remote peer 0e3bd778cdc54908bfd908adb74b0d0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:07:38.746484 21089 raft_consensus.cc:1275] T af7b5d27345443d4ba1889f61e3a1ec0 P 30fbafdb859f4ccda011013c645c20c4 [term 1 FOLLOWER]: Refusing update from remote peer 0e3bd778cdc54908bfd908adb74b0d0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:07:38.746865 21453 consensus_queue.cc:1048] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:07:38.747000 21448 consensus_queue.cc:1048] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:07:38.749580 21458 mvcc.cc:204] Tried to move back new op lower bound from 7249219005422759936 to 7249219005232484352. Current Snapshot: MvccSnapshot[applied={T|T < 7249219005422759936}]
I20260131 03:07:38.750381 21457 mvcc.cc:204] Tried to move back new op lower bound from 7249219005422759936 to 7249219005232484352. Current Snapshot: MvccSnapshot[applied={T|T < 7249219005422759936 or (T in {7249219005422759936})}]
I20260131 03:07:38.752821 20930 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:07:38.752926 20930 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:07:38.753652 20930 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146: Sending DeleteTablet for 3 replicas of tablet af7b5d27345443d4ba1889f61e3a1ec0
I20260131 03:07:38.753947 21211 tablet_service.cc:1552] Processing DeleteTablet for tablet af7b5d27345443d4ba1889f61e3a1ec0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:07:38 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:57564
I20260131 03:07:38.754017 21069 tablet_service.cc:1552] Processing DeleteTablet for tablet af7b5d27345443d4ba1889f61e3a1ec0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:07:38 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:52800
I20260131 03:07:38.754084 21353 tablet_service.cc:1552] Processing DeleteTablet for tablet af7b5d27345443d4ba1889f61e3a1ec0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:07:38 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:50558
I20260131 03:07:38.754213 21468 tablet_replica.cc:333] T af7b5d27345443d4ba1889f61e3a1ec0 P 30fbafdb859f4ccda011013c645c20c4: stopping tablet replica
I20260131 03:07:38.754284 21469 tablet_replica.cc:333] T af7b5d27345443d4ba1889f61e3a1ec0 P 9de879312e9b477d977409365f1425d1: stopping tablet replica
I20260131 03:07:38.754297 21468 raft_consensus.cc:2243] T af7b5d27345443d4ba1889f61e3a1ec0 P 30fbafdb859f4ccda011013c645c20c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:07:38.754357 21469 raft_consensus.cc:2243] T af7b5d27345443d4ba1889f61e3a1ec0 P 9de879312e9b477d977409365f1425d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:07:38.754390 21468 raft_consensus.cc:2272] T af7b5d27345443d4ba1889f61e3a1ec0 P 30fbafdb859f4ccda011013c645c20c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:07:38.754441 21469 raft_consensus.cc:2272] T af7b5d27345443d4ba1889f61e3a1ec0 P 9de879312e9b477d977409365f1425d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:07:38.754745 20930 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260131 03:07:38.754823 21468 ts_tablet_manager.cc:1916] T af7b5d27345443d4ba1889f61e3a1ec0 P 30fbafdb859f4ccda011013c645c20c4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260131 03:07:38.754905 20930 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:07:38.754978 21469 ts_tablet_manager.cc:1916] T af7b5d27345443d4ba1889f61e3a1ec0 P 9de879312e9b477d977409365f1425d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:07:38.756345 21353 tablet_service.cc:1505] Processing CreateTablet for tablet 3387fad0176d4a82af9601c20f8d3f75 (DEFAULT_TABLE table=test-table [id=50d2d4fe461a47129c33fb731bafef97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:07:38.756459 21353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3387fad0176d4a82af9601c20f8d3f75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:38.756464 21468 ts_tablet_manager.cc:1929] T af7b5d27345443d4ba1889f61e3a1ec0 P 30fbafdb859f4ccda011013c645c20c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:07:38.756544 21468 log.cc:1199] T af7b5d27345443d4ba1889f61e3a1ec0 P 30fbafdb859f4ccda011013c645c20c4: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-0/wal/wals/af7b5d27345443d4ba1889f61e3a1ec0
I20260131 03:07:38.756623 21469 ts_tablet_manager.cc:1929] T af7b5d27345443d4ba1889f61e3a1ec0 P 9de879312e9b477d977409365f1425d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:07:38.756681 21469 log.cc:1199] T af7b5d27345443d4ba1889f61e3a1ec0 P 9de879312e9b477d977409365f1425d1: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/af7b5d27345443d4ba1889f61e3a1ec0
I20260131 03:07:38.756824 21468 ts_tablet_manager.cc:1950] T af7b5d27345443d4ba1889f61e3a1ec0 P 30fbafdb859f4ccda011013c645c20c4: Deleting consensus metadata
I20260131 03:07:38.756937 21469 ts_tablet_manager.cc:1950] T af7b5d27345443d4ba1889f61e3a1ec0 P 9de879312e9b477d977409365f1425d1: Deleting consensus metadata
I20260131 03:07:38.757368 20916 catalog_manager.cc:4985] TS 9de879312e9b477d977409365f1425d1 (127.19.84.3:36581): tablet af7b5d27345443d4ba1889f61e3a1ec0 (table test-table [id=9441163cdf6f41b1975ace6d263bf7b0]) successfully deleted
I20260131 03:07:38.757786 21442 tablet_bootstrap.cc:492] T 3387fad0176d4a82af9601c20f8d3f75 P 9de879312e9b477d977409365f1425d1: Bootstrap starting.
I20260131 03:07:38.758203 21442 tablet_bootstrap.cc:654] T 3387fad0176d4a82af9601c20f8d3f75 P 9de879312e9b477d977409365f1425d1: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:38.758188 20914 catalog_manager.cc:4985] TS 30fbafdb859f4ccda011013c645c20c4 (127.19.84.1:39417): tablet af7b5d27345443d4ba1889f61e3a1ec0 (table test-table [id=9441163cdf6f41b1975ace6d263bf7b0]) successfully deleted
I20260131 03:07:38.758284 21211 tablet_service.cc:1505] Processing CreateTablet for tablet 3387fad0176d4a82af9601c20f8d3f75 (DEFAULT_TABLE table=test-table [id=50d2d4fe461a47129c33fb731bafef97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:07:38.758392 21211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3387fad0176d4a82af9601c20f8d3f75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:38.758534 21467 tablet_replica.cc:333] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b: stopping tablet replica
I20260131 03:07:38.758642 21467 raft_consensus.cc:2243] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:07:38.758773 21467 raft_consensus.cc:2272] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:07:38.759191 21467 ts_tablet_manager.cc:1916] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:07:38.759456 21069 tablet_service.cc:1505] Processing CreateTablet for tablet 3387fad0176d4a82af9601c20f8d3f75 (DEFAULT_TABLE table=test-table [id=50d2d4fe461a47129c33fb731bafef97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:07:38.759573 21069 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3387fad0176d4a82af9601c20f8d3f75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:38.760810 21445 tablet_bootstrap.cc:492] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4: Bootstrap starting.
I20260131 03:07:38.760810 21467 ts_tablet_manager.cc:1929] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:07:38.760879 21467 log.cc:1199] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-1/wal/wals/af7b5d27345443d4ba1889f61e3a1ec0
I20260131 03:07:38.761152 21467 ts_tablet_manager.cc:1950] T af7b5d27345443d4ba1889f61e3a1ec0 P 0e3bd778cdc54908bfd908adb74b0d0b: Deleting consensus metadata
I20260131 03:07:38.761237 21445 tablet_bootstrap.cc:654] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:38.761590 20914 catalog_manager.cc:4985] TS 0e3bd778cdc54908bfd908adb74b0d0b (127.19.84.2:45829): tablet af7b5d27345443d4ba1889f61e3a1ec0 (table test-table [id=9441163cdf6f41b1975ace6d263bf7b0]) successfully deleted
I20260131 03:07:38.761960 21445 tablet_bootstrap.cc:492] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4: No bootstrap required, opened a new log
I20260131 03:07:38.762018 21445 ts_tablet_manager.cc:1403] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:38.762092 21443 tablet_bootstrap.cc:492] T 3387fad0176d4a82af9601c20f8d3f75 P 0e3bd778cdc54908bfd908adb74b0d0b: Bootstrap starting.
I20260131 03:07:38.762194 21445 raft_consensus.cc:359] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.762282 21445 raft_consensus.cc:385] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:38.762308 21445 raft_consensus.cc:740] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30fbafdb859f4ccda011013c645c20c4, State: Initialized, Role: FOLLOWER
I20260131 03:07:38.762362 21445 consensus_queue.cc:260] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.762450 21445 ts_tablet_manager.cc:1434] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:38.762531 21443 tablet_bootstrap.cc:654] T 3387fad0176d4a82af9601c20f8d3f75 P 0e3bd778cdc54908bfd908adb74b0d0b: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:38.763181 21443 tablet_bootstrap.cc:492] T 3387fad0176d4a82af9601c20f8d3f75 P 0e3bd778cdc54908bfd908adb74b0d0b: No bootstrap required, opened a new log
I20260131 03:07:38.763257 21443 ts_tablet_manager.cc:1403] T 3387fad0176d4a82af9601c20f8d3f75 P 0e3bd778cdc54908bfd908adb74b0d0b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:38.763425 21443 raft_consensus.cc:359] T 3387fad0176d4a82af9601c20f8d3f75 P 0e3bd778cdc54908bfd908adb74b0d0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.763494 21443 raft_consensus.cc:385] T 3387fad0176d4a82af9601c20f8d3f75 P 0e3bd778cdc54908bfd908adb74b0d0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:38.763510 21443 raft_consensus.cc:740] T 3387fad0176d4a82af9601c20f8d3f75 P 0e3bd778cdc54908bfd908adb74b0d0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e3bd778cdc54908bfd908adb74b0d0b, State: Initialized, Role: FOLLOWER
I20260131 03:07:38.763564 21443 consensus_queue.cc:260] T 3387fad0176d4a82af9601c20f8d3f75 P 0e3bd778cdc54908bfd908adb74b0d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.763657 21443 ts_tablet_manager.cc:1434] T 3387fad0176d4a82af9601c20f8d3f75 P 0e3bd778cdc54908bfd908adb74b0d0b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:38.764087 21442 tablet_bootstrap.cc:492] T 3387fad0176d4a82af9601c20f8d3f75 P 9de879312e9b477d977409365f1425d1: No bootstrap required, opened a new log
I20260131 03:07:38.764163 21442 ts_tablet_manager.cc:1403] T 3387fad0176d4a82af9601c20f8d3f75 P 9de879312e9b477d977409365f1425d1: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260131 03:07:38.764319 21442 raft_consensus.cc:359] T 3387fad0176d4a82af9601c20f8d3f75 P 9de879312e9b477d977409365f1425d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.764377 21442 raft_consensus.cc:385] T 3387fad0176d4a82af9601c20f8d3f75 P 9de879312e9b477d977409365f1425d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:38.764397 21442 raft_consensus.cc:740] T 3387fad0176d4a82af9601c20f8d3f75 P 9de879312e9b477d977409365f1425d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9de879312e9b477d977409365f1425d1, State: Initialized, Role: FOLLOWER
I20260131 03:07:38.764447 21442 consensus_queue.cc:260] T 3387fad0176d4a82af9601c20f8d3f75 P 9de879312e9b477d977409365f1425d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.764547 21442 ts_tablet_manager.cc:1434] T 3387fad0176d4a82af9601c20f8d3f75 P 9de879312e9b477d977409365f1425d1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:38.873732 21450 raft_consensus.cc:493] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:07:38.873847 21450 raft_consensus.cc:515] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.874204 21450 leader_election.cc:290] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e3bd778cdc54908bfd908adb74b0d0b (127.19.84.2:45829), 9de879312e9b477d977409365f1425d1 (127.19.84.3:36581)
I20260131 03:07:38.876992 21473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.874380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:36581 (local address 127.19.84.1:41187)
0131 03:07:38.874569 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:38.874581 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:38.874653 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:38.874821 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:38.874821 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:38.874824 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:07:38.875178 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.875181 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.876100 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.876102 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:38.876719 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.876721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.876817 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.876824 (+ 7us) client_negotiation.cc:769] Sending connection context
0131 03:07:38.876858 (+ 34us) client_negotiation.cc:240] Negotiation successful
0131 03:07:38.876895 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":67,"threads_started":1}
I20260131 03:07:38.877369 21439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.874606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:41187 (local address 127.19.84.3:36581)
0131 03:07:38.874663 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:38.874666 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:38.874681 (+ 15us) server_negotiation.cc:408] Connection header received
0131 03:07:38.874719 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:38.874720 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:38.874731 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:38.874801 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:07:38.875481 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.876004 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.876836 (+ 832us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.877274 (+ 438us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.877280 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:38.877299 (+ 19us) server_negotiation.cc:300] Negotiation successful
0131 03:07:38.877323 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260131 03:07:38.877578 21373 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3387fad0176d4a82af9601c20f8d3f75" candidate_uuid: "30fbafdb859f4ccda011013c645c20c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9de879312e9b477d977409365f1425d1" is_pre_election: true
I20260131 03:07:38.877668 21373 raft_consensus.cc:2468] T 3387fad0176d4a82af9601c20f8d3f75 P 9de879312e9b477d977409365f1425d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30fbafdb859f4ccda011013c645c20c4 in term 0.
I20260131 03:07:38.877887 21022 leader_election.cc:304] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30fbafdb859f4ccda011013c645c20c4, 9de879312e9b477d977409365f1425d1; no voters:
I20260131 03:07:38.878085 21450 raft_consensus.cc:2804] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:07:38.878182 21450 raft_consensus.cc:493] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:07:38.878213 21450 raft_consensus.cc:3060] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:38.878520 21474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.874380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:45829 (local address 127.19.84.1:37695)
0131 03:07:38.874664 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:38.874674 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:38.874751 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:38.874948 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:38.874948 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:38.874950 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:07:38.875178 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.875180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.876124 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.876125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:38.877695 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.877699 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.878320 (+ 621us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.878325 (+ 5us) client_negotiation.cc:769] Sending connection context
0131 03:07:38.878393 (+ 68us) client_negotiation.cc:240] Negotiation successful
0131 03:07:38.878433 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":121,"thread_start_us":86,"threads_started":1}
I20260131 03:07:38.878542 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.874455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:37695 (local address 127.19.84.2:45829)
0131 03:07:38.874506 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:38.874508 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:38.874726 (+ 218us) server_negotiation.cc:408] Connection header received
0131 03:07:38.874835 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:38.874836 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:38.874846 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:38.874878 (+ 32us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:07:38.875268 (+ 390us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.876012 (+ 744us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.877806 (+ 1794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.878279 (+ 473us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.878287 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:38.878426 (+ 139us) server_negotiation.cc:300] Negotiation successful
0131 03:07:38.878480 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260131 03:07:38.879142 21450 raft_consensus.cc:515] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.879318 21450 leader_election.cc:290] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4 [CANDIDATE]: Term 1 election: Requested vote from peers 0e3bd778cdc54908bfd908adb74b0d0b (127.19.84.2:45829), 9de879312e9b477d977409365f1425d1 (127.19.84.3:36581)
I20260131 03:07:38.879613 21231 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3387fad0176d4a82af9601c20f8d3f75" candidate_uuid: "30fbafdb859f4ccda011013c645c20c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" is_pre_election: true
I20260131 03:07:38.879721 21373 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3387fad0176d4a82af9601c20f8d3f75" candidate_uuid: "30fbafdb859f4ccda011013c645c20c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9de879312e9b477d977409365f1425d1"
I20260131 03:07:38.879817 21373 raft_consensus.cc:3060] T 3387fad0176d4a82af9601c20f8d3f75 P 9de879312e9b477d977409365f1425d1 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:38.879684 21230 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3387fad0176d4a82af9601c20f8d3f75" candidate_uuid: "30fbafdb859f4ccda011013c645c20c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e3bd778cdc54908bfd908adb74b0d0b"
I20260131 03:07:38.879848 21231 raft_consensus.cc:2468] T 3387fad0176d4a82af9601c20f8d3f75 P 0e3bd778cdc54908bfd908adb74b0d0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30fbafdb859f4ccda011013c645c20c4 in term 0.
I20260131 03:07:38.880647 21373 raft_consensus.cc:2468] T 3387fad0176d4a82af9601c20f8d3f75 P 9de879312e9b477d977409365f1425d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30fbafdb859f4ccda011013c645c20c4 in term 1.
I20260131 03:07:38.880800 21022 leader_election.cc:304] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30fbafdb859f4ccda011013c645c20c4, 9de879312e9b477d977409365f1425d1; no voters: 0e3bd778cdc54908bfd908adb74b0d0b
I20260131 03:07:38.881047 21450 raft_consensus.cc:2804] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:07:38.881251 21450 raft_consensus.cc:697] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4 [term 1 LEADER]: Becoming Leader. State: Replica: 30fbafdb859f4ccda011013c645c20c4, State: Running, Role: LEADER
I20260131 03:07:38.881347 21450 consensus_queue.cc:237] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.882125 20929 catalog_manager.cc:5654] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30fbafdb859f4ccda011013c645c20c4 (127.19.84.1). New cstate: current_term: 1 leader_uuid: "30fbafdb859f4ccda011013c645c20c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } health_report { overall_health: UNKNOWN } } }
Jan 31 03:07:38 dist-test-slave-5nnp krb5kdc[20882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828857, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:38.943826 21483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.936216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:45499 (local address 127.0.0.1:44690)
0131 03:07:38.936377 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:38.936389 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:38.936470 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:38.936709 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:38.936710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:38.937107 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:38.937281 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.937283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.938048 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.938049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:38.938430 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.938432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.938707 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.939308 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:38.939313 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:38.941048 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:38.942787 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:38.942795 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:38.942797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:38.942992 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:38.943167 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:38.943168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:38.943168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:38.943194 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:38.943490 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:38.943497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:38.943580 (+ 83us) client_negotiation.cc:769] Sending connection context
0131 03:07:38.943653 (+ 73us) client_negotiation.cc:240] Negotiation successful
0131 03:07:38.943745 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260131 03:07:38.943961 21484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.936304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44690 (local address 127.19.84.62:45499)
0131 03:07:38.936507 (+ 203us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:38.936509 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:38.936558 (+ 49us) server_negotiation.cc:408] Connection header received
0131 03:07:38.936583 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:38.936584 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:38.936594 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:38.936666 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:38.937378 (+ 712us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.937943 (+ 565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.938705 (+ 762us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.938848 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.941146 (+ 2298us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:38.941151 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:38.941151 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:38.941158 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:38.942698 (+ 1540us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:38.943076 (+ 378us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:38.943077 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:38.943078 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:38.943103 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:38.943267 (+ 164us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:38.943268 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:38.943269 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:38.943413 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:38.943488 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:38.943749 (+ 261us) server_negotiation.cc:300] Negotiation successful
0131 03:07:38.943860 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":95,"threads_started":1}
I20260131 03:07:38.946167 20929 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:44690:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260131 03:07:38.946365 20929 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:07:38.948174 21211 tablet_service.cc:1505] Processing CreateTablet for tablet 46b52adaa7184c82bd52b4dcc2a5007c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f45ffdda03b4481387925c15c89d9ed8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260131 03:07:38.948179 21069 tablet_service.cc:1505] Processing CreateTablet for tablet 46b52adaa7184c82bd52b4dcc2a5007c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f45ffdda03b4481387925c15c89d9ed8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260131 03:07:38.948349 21211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46b52adaa7184c82bd52b4dcc2a5007c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:38.948361 21069 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46b52adaa7184c82bd52b4dcc2a5007c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:38.949635 21445 tablet_bootstrap.cc:492] T 46b52adaa7184c82bd52b4dcc2a5007c P 30fbafdb859f4ccda011013c645c20c4: Bootstrap starting.
I20260131 03:07:38.949693 21443 tablet_bootstrap.cc:492] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b: Bootstrap starting.
I20260131 03:07:38.949728 21353 tablet_service.cc:1505] Processing CreateTablet for tablet 46b52adaa7184c82bd52b4dcc2a5007c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f45ffdda03b4481387925c15c89d9ed8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260131 03:07:38.949859 21353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46b52adaa7184c82bd52b4dcc2a5007c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:38.950044 21443 tablet_bootstrap.cc:654] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:38.950151 21445 tablet_bootstrap.cc:654] T 46b52adaa7184c82bd52b4dcc2a5007c P 30fbafdb859f4ccda011013c645c20c4: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:38.950706 21443 tablet_bootstrap.cc:492] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b: No bootstrap required, opened a new log
I20260131 03:07:38.950775 21443 ts_tablet_manager.cc:1403] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:38.950917 21443 raft_consensus.cc:359] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.950994 21443 raft_consensus.cc:385] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:38.951022 21443 raft_consensus.cc:740] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e3bd778cdc54908bfd908adb74b0d0b, State: Initialized, Role: FOLLOWER
I20260131 03:07:38.951015 21442 tablet_bootstrap.cc:492] T 46b52adaa7184c82bd52b4dcc2a5007c P 9de879312e9b477d977409365f1425d1: Bootstrap starting.
I20260131 03:07:38.951066 21443 consensus_queue.cc:260] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.951169 21443 ts_tablet_manager.cc:1434] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260131 03:07:38.951186 21448 tablet_replica.cc:442] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } } }
I20260131 03:07:38.951251 21448 tablet_replica.cc:445] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:07:38.951488 21442 tablet_bootstrap.cc:654] T 46b52adaa7184c82bd52b4dcc2a5007c P 9de879312e9b477d977409365f1425d1: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:38.951728 21445 tablet_bootstrap.cc:492] T 46b52adaa7184c82bd52b4dcc2a5007c P 30fbafdb859f4ccda011013c645c20c4: No bootstrap required, opened a new log
I20260131 03:07:38.951776 21445 ts_tablet_manager.cc:1403] T 46b52adaa7184c82bd52b4dcc2a5007c P 30fbafdb859f4ccda011013c645c20c4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:07:38.951965 21445 raft_consensus.cc:359] T 46b52adaa7184c82bd52b4dcc2a5007c P 30fbafdb859f4ccda011013c645c20c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.952033 21445 raft_consensus.cc:385] T 46b52adaa7184c82bd52b4dcc2a5007c P 30fbafdb859f4ccda011013c645c20c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:38.952057 21445 raft_consensus.cc:740] T 46b52adaa7184c82bd52b4dcc2a5007c P 30fbafdb859f4ccda011013c645c20c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30fbafdb859f4ccda011013c645c20c4, State: Initialized, Role: FOLLOWER
I20260131 03:07:38.952122 21445 consensus_queue.cc:260] T 46b52adaa7184c82bd52b4dcc2a5007c P 30fbafdb859f4ccda011013c645c20c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.952226 21445 ts_tablet_manager.cc:1434] T 46b52adaa7184c82bd52b4dcc2a5007c P 30fbafdb859f4ccda011013c645c20c4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:38.952258 21450 tablet_replica.cc:442] T 46b52adaa7184c82bd52b4dcc2a5007c P 30fbafdb859f4ccda011013c645c20c4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } } }
I20260131 03:07:38.952365 21450 tablet_replica.cc:445] T 46b52adaa7184c82bd52b4dcc2a5007c P 30fbafdb859f4ccda011013c645c20c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:07:38.952862 21442 tablet_bootstrap.cc:492] T 46b52adaa7184c82bd52b4dcc2a5007c P 9de879312e9b477d977409365f1425d1: No bootstrap required, opened a new log
I20260131 03:07:38.952915 21442 ts_tablet_manager.cc:1403] T 46b52adaa7184c82bd52b4dcc2a5007c P 9de879312e9b477d977409365f1425d1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260131 03:07:38.953090 21442 raft_consensus.cc:359] T 46b52adaa7184c82bd52b4dcc2a5007c P 9de879312e9b477d977409365f1425d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.953166 21442 raft_consensus.cc:385] T 46b52adaa7184c82bd52b4dcc2a5007c P 9de879312e9b477d977409365f1425d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:38.953190 21442 raft_consensus.cc:740] T 46b52adaa7184c82bd52b4dcc2a5007c P 9de879312e9b477d977409365f1425d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9de879312e9b477d977409365f1425d1, State: Initialized, Role: FOLLOWER
I20260131 03:07:38.953248 21442 consensus_queue.cc:260] T 46b52adaa7184c82bd52b4dcc2a5007c P 9de879312e9b477d977409365f1425d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.953339 21442 ts_tablet_manager.cc:1434] T 46b52adaa7184c82bd52b4dcc2a5007c P 9de879312e9b477d977409365f1425d1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:38.953424 21449 tablet_replica.cc:442] T 46b52adaa7184c82bd52b4dcc2a5007c P 9de879312e9b477d977409365f1425d1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } } }
I20260131 03:07:38.953518 21449 tablet_replica.cc:445] T 46b52adaa7184c82bd52b4dcc2a5007c P 9de879312e9b477d977409365f1425d1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:07:38.961395 21448 raft_consensus.cc:493] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:07:38.961488 21448 raft_consensus.cc:515] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.961668 21448 leader_election.cc:290] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30fbafdb859f4ccda011013c645c20c4 (127.19.84.1:39417), 9de879312e9b477d977409365f1425d1 (127.19.84.3:36581)
I20260131 03:07:38.961854 21089 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "46b52adaa7184c82bd52b4dcc2a5007c" candidate_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30fbafdb859f4ccda011013c645c20c4" is_pre_election: true
I20260131 03:07:38.961979 21089 raft_consensus.cc:2468] T 46b52adaa7184c82bd52b4dcc2a5007c P 30fbafdb859f4ccda011013c645c20c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e3bd778cdc54908bfd908adb74b0d0b in term 0.
I20260131 03:07:38.961917 21373 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "46b52adaa7184c82bd52b4dcc2a5007c" candidate_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9de879312e9b477d977409365f1425d1" is_pre_election: true
I20260131 03:07:38.962033 21373 raft_consensus.cc:2468] T 46b52adaa7184c82bd52b4dcc2a5007c P 9de879312e9b477d977409365f1425d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e3bd778cdc54908bfd908adb74b0d0b in term 0.
I20260131 03:07:38.962133 21162 leader_election.cc:304] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e3bd778cdc54908bfd908adb74b0d0b, 30fbafdb859f4ccda011013c645c20c4; no voters:
I20260131 03:07:38.962251 21448 raft_consensus.cc:2804] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:07:38.962296 21448 raft_consensus.cc:493] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:07:38.962320 21448 raft_consensus.cc:3060] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:38.963006 21448 raft_consensus.cc:515] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.963167 21448 leader_election.cc:290] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b [CANDIDATE]: Term 1 election: Requested vote from peers 30fbafdb859f4ccda011013c645c20c4 (127.19.84.1:39417), 9de879312e9b477d977409365f1425d1 (127.19.84.3:36581)
I20260131 03:07:38.963409 21373 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "46b52adaa7184c82bd52b4dcc2a5007c" candidate_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9de879312e9b477d977409365f1425d1"
I20260131 03:07:38.963424 21089 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "46b52adaa7184c82bd52b4dcc2a5007c" candidate_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30fbafdb859f4ccda011013c645c20c4"
I20260131 03:07:38.963670 21089 raft_consensus.cc:3060] T 46b52adaa7184c82bd52b4dcc2a5007c P 30fbafdb859f4ccda011013c645c20c4 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:38.963691 21373 raft_consensus.cc:3060] T 46b52adaa7184c82bd52b4dcc2a5007c P 9de879312e9b477d977409365f1425d1 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:38.964495 21373 raft_consensus.cc:2468] T 46b52adaa7184c82bd52b4dcc2a5007c P 9de879312e9b477d977409365f1425d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e3bd778cdc54908bfd908adb74b0d0b in term 1.
I20260131 03:07:38.964495 21089 raft_consensus.cc:2468] T 46b52adaa7184c82bd52b4dcc2a5007c P 30fbafdb859f4ccda011013c645c20c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e3bd778cdc54908bfd908adb74b0d0b in term 1.
I20260131 03:07:38.964795 21164 leader_election.cc:304] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e3bd778cdc54908bfd908adb74b0d0b, 9de879312e9b477d977409365f1425d1; no voters:
I20260131 03:07:38.965034 21448 raft_consensus.cc:2804] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:07:38.965106 21448 raft_consensus.cc:697] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b [term 1 LEADER]: Becoming Leader. State: Replica: 0e3bd778cdc54908bfd908adb74b0d0b, State: Running, Role: LEADER
I20260131 03:07:38.965163 21448 consensus_queue.cc:237] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } }
I20260131 03:07:38.965478 21453 tablet_replica.cc:442] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b: TxnStatusTablet state changed. Reason: New leader 0e3bd778cdc54908bfd908adb74b0d0b. Latest consensus state: current_term: 1 leader_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } } }
I20260131 03:07:38.965550 21453 tablet_replica.cc:445] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b: This TxnStatusTablet replica's current role is: LEADER
I20260131 03:07:38.965766 21489 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260131 03:07:38.965894 21489 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260131 03:07:38.965842 20929 catalog_manager.cc:5654] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e3bd778cdc54908bfd908adb74b0d0b (127.19.84.2). New cstate: current_term: 1 leader_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } health_report { overall_health: UNKNOWN } } }
I20260131 03:07:38.975090 21483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.972811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:45829 (local address 127.0.0.1:57584)
0131 03:07:38.972924 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:38.972937 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:38.973030 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:38.973197 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:38.973198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:38.973200 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:07:38.973362 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.973363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.973981 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.973982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:38.974404 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.974405 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.974500 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.974520 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:07:38.974901 (+ 381us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:07:38.974913 (+ 12us) client_negotiation.cc:769] Sending connection context
0131 03:07:38.974965 (+ 52us) client_negotiation.cc:240] Negotiation successful
0131 03:07:38.975024 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260131 03:07:38.975090 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.972954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57584 (local address 127.19.84.2:45829)
0131 03:07:38.973019 (+ 65us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:38.973021 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:38.973031 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:07:38.973103 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:38.973104 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:38.973121 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:38.973187 (+ 66us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:07:38.973439 (+ 252us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.973896 (+ 457us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.974518 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.974753 (+ 235us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.974781 (+ 28us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:07:38.974811 (+ 30us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:07:38.974897 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:38.974981 (+ 84us) server_negotiation.cc:300] Negotiation successful
0131 03:07:38.975008 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260131 03:07:38.976186 21089 raft_consensus.cc:1275] T 46b52adaa7184c82bd52b4dcc2a5007c P 30fbafdb859f4ccda011013c645c20c4 [term 1 FOLLOWER]: Refusing update from remote peer 0e3bd778cdc54908bfd908adb74b0d0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:07:38.976238 21373 raft_consensus.cc:1275] T 46b52adaa7184c82bd52b4dcc2a5007c P 9de879312e9b477d977409365f1425d1 [term 1 FOLLOWER]: Refusing update from remote peer 0e3bd778cdc54908bfd908adb74b0d0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:07:38.976419 21448 consensus_queue.cc:1048] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:07:38.976528 21453 consensus_queue.cc:1048] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:07:38.977200 21450 tablet_replica.cc:442] T 46b52adaa7184c82bd52b4dcc2a5007c P 30fbafdb859f4ccda011013c645c20c4: TxnStatusTablet state changed. Reason: New leader 0e3bd778cdc54908bfd908adb74b0d0b. Latest consensus state: current_term: 1 leader_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } } }
I20260131 03:07:38.977293 21450 tablet_replica.cc:445] T 46b52adaa7184c82bd52b4dcc2a5007c P 30fbafdb859f4ccda011013c645c20c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:07:38.977460 21449 tablet_replica.cc:442] T 46b52adaa7184c82bd52b4dcc2a5007c P 9de879312e9b477d977409365f1425d1: TxnStatusTablet state changed. Reason: New leader 0e3bd778cdc54908bfd908adb74b0d0b. Latest consensus state: current_term: 1 leader_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } } }
I20260131 03:07:38.977537 21449 tablet_replica.cc:445] T 46b52adaa7184c82bd52b4dcc2a5007c P 9de879312e9b477d977409365f1425d1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:07:38.978068 21448 tablet_replica.cc:442] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } } }
I20260131 03:07:38.978178 21448 tablet_replica.cc:445] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b: This TxnStatusTablet replica's current role is: LEADER
I20260131 03:07:38.978267 21456 mvcc.cc:204] Tried to move back new op lower bound from 7249219006364565504 to 7249219006321655808. Current Snapshot: MvccSnapshot[applied={T|T < 7249219006364565504 or (T in {7249219006364565504})}]
I20260131 03:07:38.978399 21461 tablet_replica.cc:442] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } } }
I20260131 03:07:38.978464 21461 tablet_replica.cc:445] T 46b52adaa7184c82bd52b4dcc2a5007c P 0e3bd778cdc54908bfd908adb74b0d0b: This TxnStatusTablet replica's current role is: LEADER
I20260131 03:07:38.978763 21449 tablet_replica.cc:442] T 46b52adaa7184c82bd52b4dcc2a5007c P 9de879312e9b477d977409365f1425d1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } } }
I20260131 03:07:38.978842 21449 tablet_replica.cc:445] T 46b52adaa7184c82bd52b4dcc2a5007c P 9de879312e9b477d977409365f1425d1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:07:38.979741 21450 tablet_replica.cc:442] T 46b52adaa7184c82bd52b4dcc2a5007c P 30fbafdb859f4ccda011013c645c20c4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30fbafdb859f4ccda011013c645c20c4" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39417 } } peers { permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 } } peers { permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 } } }
I20260131 03:07:38.979852 21450 tablet_replica.cc:445] T 46b52adaa7184c82bd52b4dcc2a5007c P 30fbafdb859f4ccda011013c645c20c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:07:38.985445 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.982587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52812 (local address 127.19.84.1:39417)
0131 03:07:38.982686 (+ 99us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:38.982689 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:38.982700 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:07:38.982739 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:38.982740 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:38.982760 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:38.982860 (+ 100us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:07:38.983191 (+ 331us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.983846 (+ 655us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.984670 (+ 824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.984888 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.984932 (+ 44us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:07:38.985057 (+ 125us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:07:38.985135 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:38.985327 (+ 192us) server_negotiation.cc:300] Negotiation successful
0131 03:07:38.985357 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260131 03:07:38.990908 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.988417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:60207 (local address 127.19.84.2:45829)
0131 03:07:38.988489 (+ 72us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:38.988496 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:38.988638 (+ 142us) server_negotiation.cc:408] Connection header received
0131 03:07:38.988669 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:38.988669 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:38.988680 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:38.988739 (+ 59us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:07:38.989096 (+ 357us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.989589 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.990445 (+ 856us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.990575 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.990601 (+ 26us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:07:38.990623 (+ 22us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:07:38.990674 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:38.990826 (+ 152us) server_negotiation.cc:300] Negotiation successful
0131 03:07:38.990852 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260131 03:07:38.990921 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:38.988301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:45829 (local address 127.19.84.1:60207)
0131 03:07:38.988494 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:38.988504 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:38.988569 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:38.988761 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:38.988762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:38.988764 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:07:38.989009 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.989010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.989666 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:38.989667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:38.990344 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:38.990346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:38.990440 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:38.990462 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:07:38.990697 (+ 235us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:07:38.990699 (+ 2us) client_negotiation.cc:769] Sending connection context
0131 03:07:38.990749 (+ 50us) client_negotiation.cc:240] Negotiation successful
0131 03:07:38.990797 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":94,"threads_started":1}
I20260131 03:07:38.993582 21230 raft_consensus.cc:3060] T 3387fad0176d4a82af9601c20f8d3f75 P 0e3bd778cdc54908bfd908adb74b0d0b [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:38.994334 21230 raft_consensus.cc:1275] T 3387fad0176d4a82af9601c20f8d3f75 P 0e3bd778cdc54908bfd908adb74b0d0b [term 1 FOLLOWER]: Refusing update from remote peer 30fbafdb859f4ccda011013c645c20c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:07:38.994516 21373 raft_consensus.cc:1275] T 3387fad0176d4a82af9601c20f8d3f75 P 9de879312e9b477d977409365f1425d1 [term 1 FOLLOWER]: Refusing update from remote peer 30fbafdb859f4ccda011013c645c20c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:07:38.994671 21475 consensus_queue.cc:1048] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e3bd778cdc54908bfd908adb74b0d0b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:07:38.994902 21450 consensus_queue.cc:1048] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9de879312e9b477d977409365f1425d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:07:39.003749 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:39.001220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:33083 (local address 127.19.84.1:39417)
0131 03:07:39.001260 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:39.001263 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:39.001404 (+ 141us) server_negotiation.cc:408] Connection header received
0131 03:07:39.001510 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:39.001512 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:39.001526 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:39.001581 (+ 55us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:07:39.002016 (+ 435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:39.002473 (+ 457us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:39.003091 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:39.003311 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:39.003335 (+ 24us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:07:39.003364 (+ 29us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:07:39.003458 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:39.003639 (+ 181us) server_negotiation.cc:300] Negotiation successful
0131 03:07:39.003667 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260131 03:07:39.003751 21499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:39.001057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:39417 (local address 127.19.84.2:33083)
0131 03:07:39.001343 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:39.001353 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:39.001428 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:39.001642 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:39.001643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:39.001646 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:07:39.001933 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:39.001935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:39.002556 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:39.002557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:39.003004 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:39.003006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:39.003094 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:39.003115 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:07:39.003536 (+ 421us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:07:39.003538 (+ 2us) client_negotiation.cc:769] Sending connection context
0131 03:07:39.003631 (+ 93us) client_negotiation.cc:240] Negotiation successful
0131 03:07:39.003681 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":78,"threads_started":1}
I20260131 03:07:39.015864 20930 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:07:39.016094 20930 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:07:39.017046 20930 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 3f2e39897d584137a914e45c61e37146: Sending DeleteTablet for 3 replicas of tablet 3387fad0176d4a82af9601c20f8d3f75
I20260131 03:07:39.017431 21353 tablet_service.cc:1552] Processing DeleteTablet for tablet 3387fad0176d4a82af9601c20f8d3f75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:07:39 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:50558
I20260131 03:07:39.017553 21069 tablet_service.cc:1552] Processing DeleteTablet for tablet 3387fad0176d4a82af9601c20f8d3f75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:07:39 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:52800
I20260131 03:07:39.017599 21469 tablet_replica.cc:333] T 3387fad0176d4a82af9601c20f8d3f75 P 9de879312e9b477d977409365f1425d1: stopping tablet replica
I20260131 03:07:39.017650 21469 raft_consensus.cc:2243] T 3387fad0176d4a82af9601c20f8d3f75 P 9de879312e9b477d977409365f1425d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:07:39.017655 21468 tablet_replica.cc:333] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4: stopping tablet replica
I20260131 03:07:39.017699 21469 raft_consensus.cc:2272] T 3387fad0176d4a82af9601c20f8d3f75 P 9de879312e9b477d977409365f1425d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:07:39.017721 21468 raft_consensus.cc:2243] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:07:39.017805 21468 raft_consensus.cc:2272] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:07:39.018148 21469 ts_tablet_manager.cc:1916] T 3387fad0176d4a82af9601c20f8d3f75 P 9de879312e9b477d977409365f1425d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:07:39.018164 21468 ts_tablet_manager.cc:1916] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:07:39.018606 21211 tablet_service.cc:1552] Processing DeleteTablet for tablet 3387fad0176d4a82af9601c20f8d3f75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:07:39 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:57564
I20260131 03:07:39.019477 21467 tablet_replica.cc:333] T 3387fad0176d4a82af9601c20f8d3f75 P 0e3bd778cdc54908bfd908adb74b0d0b: stopping tablet replica
I20260131 03:07:39.019544 21467 raft_consensus.cc:2243] T 3387fad0176d4a82af9601c20f8d3f75 P 0e3bd778cdc54908bfd908adb74b0d0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:07:39.019586 21467 raft_consensus.cc:2272] T 3387fad0176d4a82af9601c20f8d3f75 P 0e3bd778cdc54908bfd908adb74b0d0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:07:39.020006 21468 ts_tablet_manager.cc:1929] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:07:39.020072 21468 log.cc:1199] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-0/wal/wals/3387fad0176d4a82af9601c20f8d3f75
I20260131 03:07:39.020296 21468 ts_tablet_manager.cc:1950] T 3387fad0176d4a82af9601c20f8d3f75 P 30fbafdb859f4ccda011013c645c20c4: Deleting consensus metadata
I20260131 03:07:39.020349 21469 ts_tablet_manager.cc:1929] T 3387fad0176d4a82af9601c20f8d3f75 P 9de879312e9b477d977409365f1425d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:07:39.020403 21469 log.cc:1199] T 3387fad0176d4a82af9601c20f8d3f75 P 9de879312e9b477d977409365f1425d1: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/3387fad0176d4a82af9601c20f8d3f75
I20260131 03:07:39.020404 21467 ts_tablet_manager.cc:1916] T 3387fad0176d4a82af9601c20f8d3f75 P 0e3bd778cdc54908bfd908adb74b0d0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:07:39.020624 21469 ts_tablet_manager.cc:1950] T 3387fad0176d4a82af9601c20f8d3f75 P 9de879312e9b477d977409365f1425d1: Deleting consensus metadata
I20260131 03:07:39.020679 20914 catalog_manager.cc:4985] TS 30fbafdb859f4ccda011013c645c20c4 (127.19.84.1:39417): tablet 3387fad0176d4a82af9601c20f8d3f75 (table test-table [id=50d2d4fe461a47129c33fb731bafef97]) successfully deleted
I20260131 03:07:39.020913 20916 catalog_manager.cc:4985] TS 9de879312e9b477d977409365f1425d1 (127.19.84.3:36581): tablet 3387fad0176d4a82af9601c20f8d3f75 (table test-table [id=50d2d4fe461a47129c33fb731bafef97]) successfully deleted
I20260131 03:07:39.021848 21467 ts_tablet_manager.cc:1929] T 3387fad0176d4a82af9601c20f8d3f75 P 0e3bd778cdc54908bfd908adb74b0d0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:07:39.021927 21467 log.cc:1199] T 3387fad0176d4a82af9601c20f8d3f75 P 0e3bd778cdc54908bfd908adb74b0d0b: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769828849314684-19792-0/minicluster-data/ts-1/wal/wals/3387fad0176d4a82af9601c20f8d3f75
I20260131 03:07:39.022117 21467 ts_tablet_manager.cc:1950] T 3387fad0176d4a82af9601c20f8d3f75 P 0e3bd778cdc54908bfd908adb74b0d0b: Deleting consensus metadata
I20260131 03:07:39.022796 20914 catalog_manager.cc:4985] TS 0e3bd778cdc54908bfd908adb74b0d0b (127.19.84.2:45829): tablet 3387fad0176d4a82af9601c20f8d3f75 (table test-table [id=50d2d4fe461a47129c33fb731bafef97]) successfully deleted
Jan 31 03:07:39 dist-test-slave-5nnp krb5kdc[20882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828858, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.84.1@KRBTEST.COM
I20260131 03:07:39.031251 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:39.019808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52820 (local address 127.19.84.1:39417)
0131 03:07:39.020114 (+ 306us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:39.020116 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:39.020127 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:07:39.020151 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:39.020152 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:39.020163 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:39.020223 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:39.024272 (+ 4049us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:39.024782 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:39.025511 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:39.025770 (+ 259us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:39.028062 (+ 2292us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:39.028074 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:39.028074 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:39.028082 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:39.029849 (+ 1767us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:39.030231 (+ 382us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:39.030235 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:39.030235 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:39.030274 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:39.030448 (+ 174us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:39.030448 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:39.030449 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:39.030756 (+ 307us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:39.030969 (+ 213us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:39.031018 (+ 49us) server_negotiation.cc:300] Negotiation successful
0131 03:07:39.031124 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286}
W20260131 03:07:39.031519 21127 server_base.cc:1142] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52820
I20260131 03:07:39.042884 21484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:39.035799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44706 (local address 127.19.84.62:45499)
0131 03:07:39.035867 (+ 68us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:39.035870 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:39.035973 (+ 103us) server_negotiation.cc:408] Connection header received
0131 03:07:39.036078 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:39.036079 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:39.036104 (+ 25us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:39.036171 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:39.036972 (+ 801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:39.037541 (+ 569us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:39.038190 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:39.038982 (+ 792us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:39.039985 (+ 1003us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:39.039992 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:39.039992 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:39.040005 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:39.041657 (+ 1652us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:39.042054 (+ 397us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:39.042055 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:39.042056 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:39.042093 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:39.042275 (+ 182us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:39.042276 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:39.042277 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:39.042391 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:39.042516 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:39.042652 (+ 136us) server_negotiation.cc:300] Negotiation successful
0131 03:07:39.042760 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260131 03:07:39.043279 20930 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44706
I20260131 03:07:39.044143 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 21004
I20260131 03:07:39.050426 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 21146
I20260131 03:07:39.056533 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 21288
I20260131 03:07:39.061261 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 20898
2026-01-31T03:07:39Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3582 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:07:39 dist-test-slave-5nnp krb5kdc[21509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:07:39 dist-test-slave-5nnp krb5kdc[21509](info): set up 2 sockets
Jan 31 03:07:39 dist-test-slave-5nnp krb5kdc[21509](info): commencing operation
krb5kdc: starting...
W20260131 03:07:41.096451 19792 mini_kdc.cc:121] Time spent starting KDC: real 2.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:07:41 dist-test-slave-5nnp krb5kdc[21509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:07:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:07:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:41.247900 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:33359
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:33359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:41.330376 21525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:41.330659 21525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:41.330724 21525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:41.332288 21525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:07:41.332360 21525 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260131 03:07:41.332381 21525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:41.332401 21525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:41.332428 21525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:07:41.332459 21525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:07:41.334095 21525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:33359
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:33359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:41.334376 21525 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:41.334596 21525 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:41.337327 21530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:41.337502 21525 server_base.cc:1060] running on GCE node
W20260131 03:07:41.337657 21533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:41.337726 21531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:41.338107 21525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:41.338394 21525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:41.339553 21525 hybrid_clock.cc:648] HybridClock initialized: now 1769828861339536 us; error 52 us; skew 500 ppm
Jan 31 03:07:41 dist-test-slave-5nnp krb5kdc[21509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828861, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:41.341925 21525 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:07:41.342530 21525 webserver.cc:492] Webserver started at http://127.19.84.62:37267/ using document root <none> and password file <none>
I20260131 03:07:41.342756 21525 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:41.342811 21525 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:41.343079 21525 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:41.344281 21525 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "11a07d3643e3429591c371e578821537"
format_stamp: "Formatted at 2026-01-31 03:07:41 on dist-test-slave-5nnp"
server_key: "2169ae6164811b14e59912ea731f1a26"
server_key_iv: "363372b12e53a4c86bdf05e08e8f577e"
server_key_version: "encryptionkey@0"
I20260131 03:07:41.344619 21525 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "11a07d3643e3429591c371e578821537"
format_stamp: "Formatted at 2026-01-31 03:07:41 on dist-test-slave-5nnp"
server_key: "2169ae6164811b14e59912ea731f1a26"
server_key_iv: "363372b12e53a4c86bdf05e08e8f577e"
server_key_version: "encryptionkey@0"
I20260131 03:07:41.345943 21525 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260131 03:07:41.346801 21540 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:41.347044 21525 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:07:41.347142 21525 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "11a07d3643e3429591c371e578821537"
format_stamp: "Formatted at 2026-01-31 03:07:41 on dist-test-slave-5nnp"
server_key: "2169ae6164811b14e59912ea731f1a26"
server_key_iv: "363372b12e53a4c86bdf05e08e8f577e"
server_key_version: "encryptionkey@0"
I20260131 03:07:41.347229 21525 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:41.364193 21525 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:41.366385 21525 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:41.366680 21525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:41.372404 21525 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:33359
I20260131 03:07:41.372470 21602 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:33359 every 8 connection(s)
I20260131 03:07:41.372848 21525 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:07:41.373682 21603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:41.375774 21603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537: Bootstrap starting.
I20260131 03:07:41.376294 21603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:41.376493 21603 log.cc:826] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537: Log is configured to *not* fsync() on all Append() calls
I20260131 03:07:41.377274 21603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537: No bootstrap required, opened a new log
I20260131 03:07:41.378472 21603 raft_consensus.cc:359] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a07d3643e3429591c371e578821537" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33359 } }
I20260131 03:07:41.378602 21603 raft_consensus.cc:385] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:41.378623 21603 raft_consensus.cc:740] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11a07d3643e3429591c371e578821537, State: Initialized, Role: FOLLOWER
I20260131 03:07:41.378712 21603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a07d3643e3429591c371e578821537" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33359 } }
I20260131 03:07:41.378759 21603 raft_consensus.cc:399] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:07:41.378780 21603 raft_consensus.cc:493] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:07:41.378804 21603 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:41.379293 21603 raft_consensus.cc:515] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a07d3643e3429591c371e578821537" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33359 } }
I20260131 03:07:41.379468 21603 leader_election.cc:304] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11a07d3643e3429591c371e578821537; no voters:
I20260131 03:07:41.379639 21603 leader_election.cc:290] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:07:41.379736 21606 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:07:41.379904 21603 sys_catalog.cc:565] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:07:41.379901 21606 raft_consensus.cc:697] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537 [term 1 LEADER]: Becoming Leader. State: Replica: 11a07d3643e3429591c371e578821537, State: Running, Role: LEADER
I20260131 03:07:41.380070 21606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a07d3643e3429591c371e578821537" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33359 } }
I20260131 03:07:41.380450 21608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11a07d3643e3429591c371e578821537. Latest consensus state: current_term: 1 leader_uuid: "11a07d3643e3429591c371e578821537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a07d3643e3429591c371e578821537" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33359 } } }
I20260131 03:07:41.380462 21607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11a07d3643e3429591c371e578821537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a07d3643e3429591c371e578821537" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33359 } } }
I20260131 03:07:41.380647 21608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537 [sys.catalog]: This master's current role is: LEADER
I20260131 03:07:41.380677 21607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537 [sys.catalog]: This master's current role is: LEADER
I20260131 03:07:41.381038 21615 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:07:41.381537 21615 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:07:41.382997 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 21525
I20260131 03:07:41.383100 21615 catalog_manager.cc:1357] Generated new cluster ID: 167b9da897044486bbfd73595ecef563
I20260131 03:07:41.383127 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:07:41.383149 21615 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:07:41.383323 19792 external_mini_cluster.cc:1468] Setting key 0b43844b4eab313ecfb338c05935300c
Jan 31 03:07:41 dist-test-slave-5nnp krb5kdc[21509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:41.393064 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:41.383954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54676 (local address 127.19.84.62:33359)
0131 03:07:41.384116 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:41.384125 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:41.384144 (+ 19us) server_negotiation.cc:408] Connection header received
0131 03:07:41.384228 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:41.384232 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:41.384268 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:41.384390 (+ 122us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:41.385187 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.385936 (+ 749us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:41.386516 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.386775 (+ 259us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:41.389226 (+ 2451us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:41.389251 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:41.389252 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:41.389274 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:41.391499 (+ 2225us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:41.391890 (+ 391us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:41.391893 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:41.391893 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:41.391963 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:41.392183 (+ 220us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:41.392183 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:41.392184 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:41.392461 (+ 277us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:41.392619 (+ 158us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:41.392728 (+ 109us) server_negotiation.cc:300] Negotiation successful
0131 03:07:41.392874 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260131 03:07:41.402827 21615 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:07:41.403597 21615 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:07:41.413990 21615 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537: Generated new TSK 0
I20260131 03:07:41.414188 21615 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:41.466302 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:33359
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:41.547161 21631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:41.547377 21631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:41.547465 21631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:41.548931 21631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:41.549005 21631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:41.549041 21631 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260131 03:07:41.549081 21631 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260131 03:07:41.549117 21631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:07:41.550724 21631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.84.62:33359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:41.550966 21631 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:41.551151 21631 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:41.553758 21636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:41.553824 21637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:41.554014 21631 server_base.cc:1060] running on GCE node
W20260131 03:07:41.553758 21639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:41.554278 21631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:41.554497 21631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:41.555642 21631 hybrid_clock.cc:648] HybridClock initialized: now 1769828861555610 us; error 48 us; skew 500 ppm
Jan 31 03:07:41 dist-test-slave-5nnp krb5kdc[21509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828861, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:41.557935 21631 init.cc:377] Logged in from keytab as kudu/127.19.84.1@KRBTEST.COM (short username kudu)
I20260131 03:07:41.558468 21631 webserver.cc:492] Webserver started at http://127.19.84.1:43641/ using document root <none> and password file <none>
I20260131 03:07:41.558634 21631 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:41.558666 21631 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:41.558744 21631 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:41.559706 21631 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "4765facb107747888bcb47b17955a2b8"
format_stamp: "Formatted at 2026-01-31 03:07:41 on dist-test-slave-5nnp"
server_key: "011a61160999a81cacfd81e239ed920c"
server_key_iv: "f27d7e0affe510325d494c553809c514"
server_key_version: "encryptionkey@0"
I20260131 03:07:41.559966 21631 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "4765facb107747888bcb47b17955a2b8"
format_stamp: "Formatted at 2026-01-31 03:07:41 on dist-test-slave-5nnp"
server_key: "011a61160999a81cacfd81e239ed920c"
server_key_iv: "f27d7e0affe510325d494c553809c514"
server_key_version: "encryptionkey@0"
I20260131 03:07:41.561080 21631 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:41.561712 21646 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:41.561939 21631 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:41.561995 21631 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "4765facb107747888bcb47b17955a2b8"
format_stamp: "Formatted at 2026-01-31 03:07:41 on dist-test-slave-5nnp"
server_key: "011a61160999a81cacfd81e239ed920c"
server_key_iv: "f27d7e0affe510325d494c553809c514"
server_key_version: "encryptionkey@0"
I20260131 03:07:41.562043 21631 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:41.580768 21631 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:41.582871 21631 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:41.583007 21631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:41.583606 21631 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:07:41.583647 21631 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:41.583671 21631 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:07:41.583684 21631 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 31 03:07:41 dist-test-slave-5nnp krb5kdc[21509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828861, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:41.590965 21631 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:46445
I20260131 03:07:41.591099 21761 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:46445 every 8 connection(s)
I20260131 03:07:41.591410 21631 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
I20260131 03:07:41.592437 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 21631
I20260131 03:07:41.592551 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:07:41.592756 19792 external_mini_cluster.cc:1468] Setting key 2b304b3c23b3823686d7abc813c7b826
I20260131 03:07:41.593456 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:41.584055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:36651 (local address 127.19.84.62:33359)
0131 03:07:41.584131 (+ 76us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:41.584133 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:41.584263 (+ 130us) server_negotiation.cc:408] Connection header received
0131 03:07:41.584804 (+ 541us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:41.584805 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:41.584826 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:41.584895 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:41.585892 (+ 997us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.586760 (+ 868us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:41.587613 (+ 853us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.587801 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:41.590380 (+ 2579us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:41.590385 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:41.590386 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:41.590395 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:41.592154 (+ 1759us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:41.592496 (+ 342us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:41.592497 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:41.592498 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:41.592530 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:41.592777 (+ 247us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:41.592779 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:41.592779 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:41.592933 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:41.592996 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:41.593286 (+ 290us) server_negotiation.cc:300] Negotiation successful
0131 03:07:41.593391 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260131 03:07:41.593616 21655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:41.583986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33359 (local address 127.19.84.1:36651)
0131 03:07:41.584157 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:41.584187 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:41.584684 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:41.584957 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:41.584961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:41.585318 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:41.585794 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:41.585798 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.586865 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:41.586866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:41.587530 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:41.587533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.587660 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:41.588186 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:41.588210 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:41.590271 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:41.592226 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:41.592230 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:41.592232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:41.592421 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:41.592618 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:41.592619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:41.592620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:41.592709 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:41.593006 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:41.593012 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:41.593159 (+ 147us) client_negotiation.cc:769] Sending connection context
0131 03:07:41.593249 (+ 90us) client_negotiation.cc:240] Negotiation successful
0131 03:07:41.593396 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":98,"threads_started":1}
I20260131 03:07:41.595067 21762 heartbeater.cc:344] Connected to a master server at 127.19.84.62:33359
I20260131 03:07:41.595152 21762 heartbeater.cc:461] Registering TS with master...
I20260131 03:07:41.595407 21762 heartbeater.cc:507] Master 127.19.84.62:33359 requested a full tablet report, sending...
I20260131 03:07:41.595925 21556 ts_manager.cc:194] Registered new tserver with Master: 4765facb107747888bcb47b17955a2b8 (127.19.84.1:46445)
I20260131 03:07:41.597045 21556 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.1@KRBTEST.COM'} at 127.19.84.1:36651
WARNING: no policy specified for kudu/127.19.84.2@KRBTEST.COM; defaulting to no policy
I20260131 03:07:41.603513 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:41.597438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:50989 (local address 127.19.84.62:33359)
0131 03:07:41.597504 (+ 66us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:41.597506 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:41.598236 (+ 730us) server_negotiation.cc:408] Connection header received
0131 03:07:41.598266 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:41.598268 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:41.598280 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:41.598348 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:41.598792 (+ 444us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.599652 (+ 860us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:41.600193 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.600375 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:41.601033 (+ 658us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:41.601037 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:41.601038 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:41.601044 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:41.602572 (+ 1528us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:41.602888 (+ 316us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:41.602889 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:41.602889 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:41.602912 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:41.603089 (+ 177us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:41.603089 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:41.603090 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:41.603166 (+ 76us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:41.603211 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:41.603319 (+ 108us) server_negotiation.cc:300] Negotiation successful
0131 03:07:41.603445 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260131 03:07:41.603574 21769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:41.597494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33359 (local address 127.19.84.1:50989)
0131 03:07:41.598035 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:41.598049 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:41.598130 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:41.598370 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:41.598371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:41.598523 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:41.598703 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:41.598705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.599735 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:41.599736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:41.600102 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:41.600103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.600164 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:41.600676 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:41.600680 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:41.600944 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:41.602649 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:41.602650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:41.602651 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:41.602810 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:41.602994 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:41.602995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:41.602996 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:41.603027 (+ 31us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:41.603217 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:41.603217 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:41.603254 (+ 37us) client_negotiation.cc:769] Sending connection context
0131 03:07:41.603306 (+ 52us) client_negotiation.cc:240] Negotiation successful
0131 03:07:41.603471 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:41.650676 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:33359
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:41.730417 21773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:41.730608 21773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:41.730633 21773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:41.732089 21773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:41.732136 21773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:41.732158 21773 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260131 03:07:41.732170 21773 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260131 03:07:41.732187 21773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:07:41.733747 21773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.84.62:33359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:41.733930 21773 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:41.734081 21773 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:41.737571 21779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:41.737629 21778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:41.737687 21773 server_base.cc:1060] running on GCE node
W20260131 03:07:41.737579 21781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:41.737990 21773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:41.738183 21773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:41.739320 21773 hybrid_clock.cc:648] HybridClock initialized: now 1769828861739306 us; error 29 us; skew 500 ppm
Jan 31 03:07:41 dist-test-slave-5nnp krb5kdc[21509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828861, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:41.741799 21773 init.cc:377] Logged in from keytab as kudu/127.19.84.2@KRBTEST.COM (short username kudu)
I20260131 03:07:41.742421 21773 webserver.cc:492] Webserver started at http://127.19.84.2:32857/ using document root <none> and password file <none>
I20260131 03:07:41.742635 21773 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:41.742698 21773 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:41.742846 21773 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:41.743950 21773 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "0e114f2373e14771981120579a59e39d"
format_stamp: "Formatted at 2026-01-31 03:07:41 on dist-test-slave-5nnp"
server_key: "e4db69600c4bf4957dd8f48e5f368a27"
server_key_iv: "6fd6e05e52ee1d64f2793aaf3da1dacd"
server_key_version: "encryptionkey@0"
I20260131 03:07:41.744297 21773 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "0e114f2373e14771981120579a59e39d"
format_stamp: "Formatted at 2026-01-31 03:07:41 on dist-test-slave-5nnp"
server_key: "e4db69600c4bf4957dd8f48e5f368a27"
server_key_iv: "6fd6e05e52ee1d64f2793aaf3da1dacd"
server_key_version: "encryptionkey@0"
I20260131 03:07:41.745601 21773 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:07:41.746378 21788 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:41.746641 21773 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:41.746743 21773 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "0e114f2373e14771981120579a59e39d"
format_stamp: "Formatted at 2026-01-31 03:07:41 on dist-test-slave-5nnp"
server_key: "e4db69600c4bf4957dd8f48e5f368a27"
server_key_iv: "6fd6e05e52ee1d64f2793aaf3da1dacd"
server_key_version: "encryptionkey@0"
I20260131 03:07:41.746827 21773 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:41.757534 21773 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:41.759681 21773 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:41.759845 21773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:41.760443 21773 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:07:41.760493 21773 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:41.760529 21773 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:07:41.760543 21773 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 31 03:07:41 dist-test-slave-5nnp krb5kdc[21509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828861, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:41.768402 21773 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:40287
I20260131 03:07:41.768543 21904 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:40287 every 8 connection(s)
I20260131 03:07:41.768882 21773 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
I20260131 03:07:41.770373 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:41.760828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:45275 (local address 127.19.84.62:33359)
0131 03:07:41.760909 (+ 81us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:41.760913 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:41.761117 (+ 204us) server_negotiation.cc:408] Connection header received
0131 03:07:41.761811 (+ 694us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:41.761813 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:41.761833 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:41.761877 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:41.763033 (+ 1156us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.763736 (+ 703us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:41.764667 (+ 931us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.764879 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:41.767575 (+ 2696us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:41.767581 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:41.767582 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:41.767590 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:41.769282 (+ 1692us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:41.769686 (+ 404us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:41.769687 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:41.769688 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:41.769712 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:41.769870 (+ 158us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:41.769871 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:41.769871 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:41.769962 (+ 91us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:41.770019 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:41.770234 (+ 215us) server_negotiation.cc:300] Negotiation successful
0131 03:07:41.770309 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260131 03:07:41.770507 21797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:41.760808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33359 (local address 127.19.84.2:45275)
0131 03:07:41.760985 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:41.761024 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:41.761713 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:41.761945 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:41.761949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:41.762454 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:41.762916 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:41.762921 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.763853 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:41.763855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:41.764550 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:41.764553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.764714 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:41.765448 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:41.765474 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:41.767445 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:41.769390 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:41.769394 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:41.769397 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:41.769603 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:41.769772 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:41.769772 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:41.769773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:41.769821 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:41.770009 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:41.770011 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:41.770130 (+ 119us) client_negotiation.cc:769] Sending connection context
0131 03:07:41.770214 (+ 84us) client_negotiation.cc:240] Negotiation successful
0131 03:07:41.770318 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":101,"threads_started":1}
I20260131 03:07:41.771135 21905 heartbeater.cc:344] Connected to a master server at 127.19.84.62:33359
I20260131 03:07:41.771209 21905 heartbeater.cc:461] Registering TS with master...
I20260131 03:07:41.771476 21905 heartbeater.cc:507] Master 127.19.84.62:33359 requested a full tablet report, sending...
I20260131 03:07:41.771896 21556 ts_manager.cc:194] Registered new tserver with Master: 0e114f2373e14771981120579a59e39d (127.19.84.2:40287)
I20260131 03:07:41.772351 21556 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.2@KRBTEST.COM'} at 127.19.84.2:45275
I20260131 03:07:41.775563 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 21773
I20260131 03:07:41.775681 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:07:41.775893 19792 external_mini_cluster.cc:1468] Setting key cef1434a2661debf57f2dea4751ca00d
I20260131 03:07:41.777599 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:41.772302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33359 (local address 127.19.84.2:47515)
0131 03:07:41.772563 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:41.772579 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:41.772633 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:41.772793 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:41.772794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:41.772989 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:41.773208 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:41.773211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.773819 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:41.773819 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:41.774179 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:41.774180 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.774237 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:41.774725 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:41.774730 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:41.774978 (+ 248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:41.776714 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:41.776715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:41.776715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:41.776867 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:41.777023 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:41.777023 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:41.777024 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:41.777046 (+ 22us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:41.777315 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:41.777316 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:41.777361 (+ 45us) client_negotiation.cc:769] Sending connection context
0131 03:07:41.777430 (+ 69us) client_negotiation.cc:240] Negotiation successful
0131 03:07:41.777517 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":131,"threads_started":1}
I20260131 03:07:41.777717 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:41.772390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:47515 (local address 127.19.84.62:33359)
0131 03:07:41.772472 (+ 82us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:41.772475 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:41.772624 (+ 149us) server_negotiation.cc:408] Connection header received
0131 03:07:41.772680 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:41.772681 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:41.772692 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:41.772745 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:41.773294 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.773749 (+ 455us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:41.774245 (+ 496us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.774349 (+ 104us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:41.775057 (+ 708us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:41.775060 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:41.775060 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:41.775066 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:41.776641 (+ 1575us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:41.776935 (+ 294us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:41.776936 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:41.776937 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:41.776962 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:41.777083 (+ 121us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:41.777083 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:41.777084 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:41.777233 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:41.777295 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:41.777477 (+ 182us) server_negotiation.cc:300] Negotiation successful
0131 03:07:41.777570 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:41.826911 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:33359
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:41.908808 21916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:41.909018 21916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:41.909055 21916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:41.910487 21916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:41.910559 21916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:41.910604 21916 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260131 03:07:41.910634 21916 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260131 03:07:41.910665 21916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:07:41.912298 21916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.84.62:33359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:41.912583 21916 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:41.912784 21916 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:41.915762 21924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:41.915764 21922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:41.915776 21921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:41.916170 21916 server_base.cc:1060] running on GCE node
I20260131 03:07:41.916338 21916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:41.916740 21916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:41.917894 21916 hybrid_clock.cc:648] HybridClock initialized: now 1769828861917877 us; error 30 us; skew 500 ppm
Jan 31 03:07:41 dist-test-slave-5nnp krb5kdc[21509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828861, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:41.920464 21916 init.cc:377] Logged in from keytab as kudu/127.19.84.3@KRBTEST.COM (short username kudu)
I20260131 03:07:41.921104 21916 webserver.cc:492] Webserver started at http://127.19.84.3:42585/ using document root <none> and password file <none>
I20260131 03:07:41.921308 21916 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:41.921363 21916 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:41.921505 21916 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:41.922485 21916 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "7a3f75adeeeb4c008b60ba490d012604"
format_stamp: "Formatted at 2026-01-31 03:07:41 on dist-test-slave-5nnp"
server_key: "3f6514952c0e8320a581ebc1b8ff3bd2"
server_key_iv: "2f652e1fd53449823010d80ebe4b1ef9"
server_key_version: "encryptionkey@0"
I20260131 03:07:41.922829 21916 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "7a3f75adeeeb4c008b60ba490d012604"
format_stamp: "Formatted at 2026-01-31 03:07:41 on dist-test-slave-5nnp"
server_key: "3f6514952c0e8320a581ebc1b8ff3bd2"
server_key_iv: "2f652e1fd53449823010d80ebe4b1ef9"
server_key_version: "encryptionkey@0"
I20260131 03:07:41.924387 21916 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:07:41.925190 21931 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:41.925418 21916 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:41.925518 21916 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "7a3f75adeeeb4c008b60ba490d012604"
format_stamp: "Formatted at 2026-01-31 03:07:41 on dist-test-slave-5nnp"
server_key: "3f6514952c0e8320a581ebc1b8ff3bd2"
server_key_iv: "2f652e1fd53449823010d80ebe4b1ef9"
server_key_version: "encryptionkey@0"
I20260131 03:07:41.925611 21916 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:41.940217 21916 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:41.942274 21916 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:41.942425 21916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:41.943032 21916 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:07:41.943090 21916 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:41.943131 21916 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:07:41.943154 21916 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 31 03:07:41 dist-test-slave-5nnp krb5kdc[21509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828861, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:41.951066 21916 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:38153
I20260131 03:07:41.951191 22046 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:38153 every 8 connection(s)
I20260131 03:07:41.951550 21916 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
I20260131 03:07:41.952718 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:41.943470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:51829 (local address 127.19.84.62:33359)
0131 03:07:41.943527 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:41.943530 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:41.943744 (+ 214us) server_negotiation.cc:408] Connection header received
0131 03:07:41.944314 (+ 570us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:41.944316 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:41.944330 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:41.944407 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:41.945276 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.945978 (+ 702us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:41.946771 (+ 793us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.946948 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:41.949465 (+ 2517us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:41.949470 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:41.949470 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:41.949478 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:41.951240 (+ 1762us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:41.951753 (+ 513us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:41.951755 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:41.951755 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:41.951795 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:41.952077 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:41.952078 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:41.952078 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:41.952192 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:41.952263 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:41.952555 (+ 292us) server_negotiation.cc:300] Negotiation successful
0131 03:07:41.952629 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260131 03:07:41.952800 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:41.943461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33359 (local address 127.19.84.3:51829)
0131 03:07:41.943611 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:41.943643 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:41.944217 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:41.944465 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:41.944468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:41.944839 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:41.945177 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:41.945181 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.946074 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:41.946075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:41.946697 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:41.946699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.946839 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:41.947453 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:41.947489 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:41.949298 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:41.951341 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:41.951346 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:41.951348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:41.951673 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:41.951887 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:41.951888 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:41.951888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:41.951972 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:41.952263 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:41.952266 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:41.952387 (+ 121us) client_negotiation.cc:769] Sending connection context
0131 03:07:41.952468 (+ 81us) client_negotiation.cc:240] Negotiation successful
0131 03:07:41.952586 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260131 03:07:41.953210 22047 heartbeater.cc:344] Connected to a master server at 127.19.84.62:33359
I20260131 03:07:41.953299 22047 heartbeater.cc:461] Registering TS with master...
I20260131 03:07:41.953697 22047 heartbeater.cc:507] Master 127.19.84.62:33359 requested a full tablet report, sending...
I20260131 03:07:41.954228 21556 ts_manager.cc:194] Registered new tserver with Master: 7a3f75adeeeb4c008b60ba490d012604 (127.19.84.3:38153)
I20260131 03:07:41.954782 21556 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:51829
I20260131 03:07:41.959339 22053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:41.954197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33359 (local address 127.19.84.3:57189)
0131 03:07:41.954344 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:41.954356 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:41.954423 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:41.954537 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:41.954537 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:41.954689 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:41.955002 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:41.955004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.955815 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:41.955816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:41.956171 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:41.956172 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.956226 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:41.956701 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:41.956705 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:41.956947 (+ 242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:41.958400 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:41.958401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:41.958401 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:41.958539 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:41.958710 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:41.958711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:41.958711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:41.958735 (+ 24us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:41.959058 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:41.959061 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:41.959106 (+ 45us) client_negotiation.cc:769] Sending connection context
0131 03:07:41.959176 (+ 70us) client_negotiation.cc:240] Negotiation successful
0131 03:07:41.959256 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260131 03:07:41.959427 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:41.954341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:57189 (local address 127.19.84.62:33359)
0131 03:07:41.954377 (+ 36us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:41.954379 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:41.954408 (+ 29us) server_negotiation.cc:408] Connection header received
0131 03:07:41.954478 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:41.954478 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:41.954485 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:41.954531 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:41.955076 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.955717 (+ 641us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:41.956253 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.956432 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:41.957020 (+ 588us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:41.957023 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:41.957023 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:41.957029 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:41.958315 (+ 1286us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:41.958625 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:41.958626 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:41.958626 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:41.958647 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:41.958833 (+ 186us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:41.958834 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:41.958834 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:41.958919 (+ 85us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:41.958983 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:41.959246 (+ 263us) server_negotiation.cc:300] Negotiation successful
0131 03:07:41.959313 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260131 03:07:41.961719 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 21916
I20260131 03:07:41.961848 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:07:41.961992 19792 external_mini_cluster.cc:1468] Setting key 154f3ebf0624a90a8fabc1eb92d511f8
I20260131 03:07:41.963518 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Jan 31 03:07:41 dist-test-slave-5nnp krb5kdc[21509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 31 03:07:41 dist-test-slave-5nnp krb5kdc[21509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:41.988241 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:41.979484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54688 (local address 127.19.84.62:33359)
0131 03:07:41.979552 (+ 68us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:41.979555 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:41.979564 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:07:41.979590 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:41.979591 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:41.979601 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:41.979658 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:41.980511 (+ 853us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.981055 (+ 544us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:41.981812 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.981991 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:41.984740 (+ 2749us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:41.984749 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:41.984750 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:41.984758 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:41.986720 (+ 1962us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:41.987104 (+ 384us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:41.987106 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:41.987106 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:41.987136 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:41.987511 (+ 375us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:41.987513 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:41.987513 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:41.987626 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:41.987714 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:41.988086 (+ 372us) server_negotiation.cc:300] Negotiation successful
0131 03:07:41.988168 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260131 03:07:41.990885 21556 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:07:41.992226 21556 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:07:41.998639 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:41.995526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:38153 (local address 127.0.0.1:46458)
0131 03:07:41.995694 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:41.995727 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:41.995806 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:41.996194 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:41.996197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:41.996200 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:07:41.996487 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:41.996489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.997497 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:41.997499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:41.998335 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:41.998337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.998412 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:41.998455 (+ 43us) client_negotiation.cc:769] Sending connection context
0131 03:07:41.998505 (+ 50us) client_negotiation.cc:240] Negotiation successful
0131 03:07:41.998545 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":55,"threads_started":1}
I20260131 03:07:41.999022 22066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:41.995527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:46445 (local address 127.0.0.1:43038)
0131 03:07:41.995871 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:41.995884 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:41.995947 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:41.996238 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:41.996239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:41.996241 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:07:41.996519 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:41.996520 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.997891 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:41.997893 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:41.998799 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:41.998801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.998872 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:41.998883 (+ 11us) client_negotiation.cc:769] Sending connection context
0131 03:07:41.998912 (+ 29us) client_negotiation.cc:240] Negotiation successful
0131 03:07:41.998951 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":6,"thread_start_us":70,"threads_started":1}
I20260131 03:07:41.999274 22064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:41.995555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:40287 (local address 127.0.0.1:38498)
0131 03:07:41.995907 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:41.995917 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:41.995967 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:41.996303 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:41.996304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:41.996306 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:07:41.996519 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:41.996520 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.997920 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:41.997921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:41.998974 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:41.998977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.999057 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:41.999061 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:07:41.999168 (+ 107us) client_negotiation.cc:240] Negotiation successful
0131 03:07:41.999210 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":102,"threads_started":1}
I20260131 03:07:41.999329 22068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:41.995713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46458 (local address 127.19.84.3:38153)
0131 03:07:41.995913 (+ 200us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:41.995921 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:41.995932 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:07:41.995975 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:41.995977 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:41.995996 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:41.996068 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:07:41.996591 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.997372 (+ 781us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:41.998433 (+ 1061us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.999111 (+ 678us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:41.999172 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:41.999203 (+ 31us) server_negotiation.cc:300] Negotiation successful
0131 03:07:41.999247 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":115,"threads_started":1}
I20260131 03:07:41.999936 22069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:41.995777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43038 (local address 127.19.84.1:46445)
0131 03:07:41.995947 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:41.995954 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:41.995968 (+ 14us) server_negotiation.cc:408] Connection header received
0131 03:07:41.996035 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:41.996037 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:41.996054 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:41.996122 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:07:41.996611 (+ 489us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.997796 (+ 1185us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:41.999163 (+ 1367us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.999756 (+ 593us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:41.999803 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:41.999843 (+ 40us) server_negotiation.cc:300] Negotiation successful
0131 03:07:41.999877 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260131 03:07:42.000077 21981 tablet_service.cc:1505] Processing CreateTablet for tablet bbdd9c829e95461997a13db4067b58c5 (DEFAULT_TABLE table=test-table [id=4ff6ab6f622f42e4ab3ce5ed065411b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:07:42.000419 22067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:41.995719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38498 (local address 127.19.84.2:40287)
0131 03:07:41.996107 (+ 388us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:41.996114 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:41.996124 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:07:41.996162 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:41.996163 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:41.996177 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:41.996237 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:07:41.996689 (+ 452us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.997839 (+ 1150us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:41.999482 (+ 1643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:41.999979 (+ 497us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:42.000231 (+ 252us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:42.000261 (+ 30us) server_negotiation.cc:300] Negotiation successful
0131 03:07:42.000321 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":80,"threads_started":1}
I20260131 03:07:42.000452 21981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbdd9c829e95461997a13db4067b58c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:42.000674 21696 tablet_service.cc:1505] Processing CreateTablet for tablet bbdd9c829e95461997a13db4067b58c5 (DEFAULT_TABLE table=test-table [id=4ff6ab6f622f42e4ab3ce5ed065411b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:07:42.000988 21696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbdd9c829e95461997a13db4067b58c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:42.000923 21839 tablet_service.cc:1505] Processing CreateTablet for tablet bbdd9c829e95461997a13db4067b58c5 (DEFAULT_TABLE table=test-table [id=4ff6ab6f622f42e4ab3ce5ed065411b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:07:42.001180 21839 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbdd9c829e95461997a13db4067b58c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:42.003208 22070 tablet_bootstrap.cc:492] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8: Bootstrap starting.
I20260131 03:07:42.003679 22071 tablet_bootstrap.cc:492] T bbdd9c829e95461997a13db4067b58c5 P 7a3f75adeeeb4c008b60ba490d012604: Bootstrap starting.
I20260131 03:07:42.004040 22070 tablet_bootstrap.cc:654] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:42.004122 22072 tablet_bootstrap.cc:492] T bbdd9c829e95461997a13db4067b58c5 P 0e114f2373e14771981120579a59e39d: Bootstrap starting.
I20260131 03:07:42.004269 22071 tablet_bootstrap.cc:654] T bbdd9c829e95461997a13db4067b58c5 P 7a3f75adeeeb4c008b60ba490d012604: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:42.004854 22070 log.cc:826] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8: Log is configured to *not* fsync() on all Append() calls
I20260131 03:07:42.004899 22072 tablet_bootstrap.cc:654] T bbdd9c829e95461997a13db4067b58c5 P 0e114f2373e14771981120579a59e39d: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:42.004943 22071 log.cc:826] T bbdd9c829e95461997a13db4067b58c5 P 7a3f75adeeeb4c008b60ba490d012604: Log is configured to *not* fsync() on all Append() calls
I20260131 03:07:42.005249 22072 log.cc:826] T bbdd9c829e95461997a13db4067b58c5 P 0e114f2373e14771981120579a59e39d: Log is configured to *not* fsync() on all Append() calls
I20260131 03:07:42.005611 22071 tablet_bootstrap.cc:492] T bbdd9c829e95461997a13db4067b58c5 P 7a3f75adeeeb4c008b60ba490d012604: No bootstrap required, opened a new log
I20260131 03:07:42.005635 22070 tablet_bootstrap.cc:492] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8: No bootstrap required, opened a new log
I20260131 03:07:42.005683 22071 ts_tablet_manager.cc:1403] T bbdd9c829e95461997a13db4067b58c5 P 7a3f75adeeeb4c008b60ba490d012604: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:07:42.005682 22070 ts_tablet_manager.cc:1403] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260131 03:07:42.006222 22072 tablet_bootstrap.cc:492] T bbdd9c829e95461997a13db4067b58c5 P 0e114f2373e14771981120579a59e39d: No bootstrap required, opened a new log
I20260131 03:07:42.006304 22072 ts_tablet_manager.cc:1403] T bbdd9c829e95461997a13db4067b58c5 P 0e114f2373e14771981120579a59e39d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:07:42.007262 22070 raft_consensus.cc:359] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } } peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } }
I20260131 03:07:42.007262 22071 raft_consensus.cc:359] T bbdd9c829e95461997a13db4067b58c5 P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } } peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } }
I20260131 03:07:42.007447 22070 raft_consensus.cc:385] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:42.007483 22070 raft_consensus.cc:740] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4765facb107747888bcb47b17955a2b8, State: Initialized, Role: FOLLOWER
I20260131 03:07:42.007477 22071 raft_consensus.cc:385] T bbdd9c829e95461997a13db4067b58c5 P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:42.007512 22071 raft_consensus.cc:740] T bbdd9c829e95461997a13db4067b58c5 P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a3f75adeeeb4c008b60ba490d012604, State: Initialized, Role: FOLLOWER
I20260131 03:07:42.007597 22071 consensus_queue.cc:260] T bbdd9c829e95461997a13db4067b58c5 P 7a3f75adeeeb4c008b60ba490d012604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } } peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } }
I20260131 03:07:42.007617 22070 consensus_queue.cc:260] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } } peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } }
I20260131 03:07:42.007874 22071 ts_tablet_manager.cc:1434] T bbdd9c829e95461997a13db4067b58c5 P 7a3f75adeeeb4c008b60ba490d012604: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:07:42.007916 22070 ts_tablet_manager.cc:1434] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:07:42.007843 22072 raft_consensus.cc:359] T bbdd9c829e95461997a13db4067b58c5 P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } } peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } }
I20260131 03:07:42.007886 22047 heartbeater.cc:499] Master 127.19.84.62:33359 was elected leader, sending a full tablet report...
I20260131 03:07:42.007997 22072 raft_consensus.cc:385] T bbdd9c829e95461997a13db4067b58c5 P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:42.008006 21762 heartbeater.cc:499] Master 127.19.84.62:33359 was elected leader, sending a full tablet report...
I20260131 03:07:42.008041 22072 raft_consensus.cc:740] T bbdd9c829e95461997a13db4067b58c5 P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e114f2373e14771981120579a59e39d, State: Initialized, Role: FOLLOWER
I20260131 03:07:42.008129 22072 consensus_queue.cc:260] T bbdd9c829e95461997a13db4067b58c5 P 0e114f2373e14771981120579a59e39d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } } peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } }
I20260131 03:07:42.008394 22072 ts_tablet_manager.cc:1434] T bbdd9c829e95461997a13db4067b58c5 P 0e114f2373e14771981120579a59e39d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:07:42.008415 21905 heartbeater.cc:499] Master 127.19.84.62:33359 was elected leader, sending a full tablet report...
W20260131 03:07:42.019598 21906 tablet.cc:2378] T bbdd9c829e95461997a13db4067b58c5 P 0e114f2373e14771981120579a59e39d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260131 03:07:42.092263 21763 tablet.cc:2378] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:07:42.187179 22077 raft_consensus.cc:493] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:07:42.187307 22077 raft_consensus.cc:515] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } } peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } }
I20260131 03:07:42.187669 22077 leader_election.cc:290] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e114f2373e14771981120579a59e39d (127.19.84.2:40287), 7a3f75adeeeb4c008b60ba490d012604 (127.19.84.3:38153)
I20260131 03:07:42.190950 22079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:42.187887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:38153 (local address 127.19.84.1:41787)
0131 03:07:42.188067 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:42.188077 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:42.188130 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:42.188318 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:42.188318 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:42.188321 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:07:42.188573 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:42.188575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.189671 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:42.189672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:42.190639 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:42.190642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.190730 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:42.190736 (+ 6us) client_negotiation.cc:769] Sending connection context
0131 03:07:42.190762 (+ 26us) client_negotiation.cc:240] Negotiation successful
0131 03:07:42.190828 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":68,"threads_started":1}
I20260131 03:07:42.190950 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:42.187886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:40287 (local address 127.19.84.1:41623)
0131 03:07:42.188039 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:42.188052 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:42.188123 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:42.188318 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:42.188318 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:42.188322 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:07:42.188598 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:42.188599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.189670 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:42.189672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:42.190688 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:42.190690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.190753 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:42.190756 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:07:42.190790 (+ 34us) client_negotiation.cc:240] Negotiation successful
0131 03:07:42.190828 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260131 03:07:42.191465 22067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:42.188004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:41623 (local address 127.19.84.2:40287)
0131 03:07:42.188073 (+ 69us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:42.188075 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:42.188105 (+ 30us) server_negotiation.cc:408] Connection header received
0131 03:07:42.188225 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:42.188226 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:42.188237 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:42.188290 (+ 53us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:07:42.188707 (+ 417us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.189523 (+ 816us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:42.190784 (+ 1261us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.191288 (+ 504us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:42.191298 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:42.191329 (+ 31us) server_negotiation.cc:300] Negotiation successful
0131 03:07:42.191399 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260131 03:07:42.191468 22068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:42.187965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:41787 (local address 127.19.84.3:38153)
0131 03:07:42.188018 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:42.188038 (+ 20us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:42.188141 (+ 103us) server_negotiation.cc:408] Connection header received
0131 03:07:42.188180 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:42.188181 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:42.188194 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:42.188262 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:07:42.188707 (+ 445us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.189507 (+ 800us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:42.190797 (+ 1290us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.191323 (+ 526us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:42.191330 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:42.191351 (+ 21us) server_negotiation.cc:300] Negotiation successful
0131 03:07:42.191410 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260131 03:07:42.191766 22001 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bbdd9c829e95461997a13db4067b58c5" candidate_uuid: "4765facb107747888bcb47b17955a2b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a3f75adeeeb4c008b60ba490d012604" is_pre_election: true
I20260131 03:07:42.191869 21859 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bbdd9c829e95461997a13db4067b58c5" candidate_uuid: "4765facb107747888bcb47b17955a2b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e114f2373e14771981120579a59e39d" is_pre_election: true
I20260131 03:07:42.191941 22001 raft_consensus.cc:2468] T bbdd9c829e95461997a13db4067b58c5 P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4765facb107747888bcb47b17955a2b8 in term 0.
I20260131 03:07:42.192059 21859 raft_consensus.cc:2468] T bbdd9c829e95461997a13db4067b58c5 P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4765facb107747888bcb47b17955a2b8 in term 0.
I20260131 03:07:42.192169 21647 leader_election.cc:304] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4765facb107747888bcb47b17955a2b8, 7a3f75adeeeb4c008b60ba490d012604; no voters:
I20260131 03:07:42.192430 22077 raft_consensus.cc:2804] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:07:42.192503 22077 raft_consensus.cc:493] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:07:42.192523 22077 raft_consensus.cc:3060] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:42.193363 22077 raft_consensus.cc:515] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } } peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } }
I20260131 03:07:42.193516 22077 leader_election.cc:290] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8 [CANDIDATE]: Term 1 election: Requested vote from peers 0e114f2373e14771981120579a59e39d (127.19.84.2:40287), 7a3f75adeeeb4c008b60ba490d012604 (127.19.84.3:38153)
I20260131 03:07:42.193737 21859 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bbdd9c829e95461997a13db4067b58c5" candidate_uuid: "4765facb107747888bcb47b17955a2b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e114f2373e14771981120579a59e39d"
I20260131 03:07:42.193747 22001 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bbdd9c829e95461997a13db4067b58c5" candidate_uuid: "4765facb107747888bcb47b17955a2b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a3f75adeeeb4c008b60ba490d012604"
I20260131 03:07:42.193830 21859 raft_consensus.cc:3060] T bbdd9c829e95461997a13db4067b58c5 P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:42.193820 22001 raft_consensus.cc:3060] T bbdd9c829e95461997a13db4067b58c5 P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:42.194559 21859 raft_consensus.cc:2468] T bbdd9c829e95461997a13db4067b58c5 P 0e114f2373e14771981120579a59e39d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4765facb107747888bcb47b17955a2b8 in term 1.
I20260131 03:07:42.194576 22001 raft_consensus.cc:2468] T bbdd9c829e95461997a13db4067b58c5 P 7a3f75adeeeb4c008b60ba490d012604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4765facb107747888bcb47b17955a2b8 in term 1.
I20260131 03:07:42.194753 21649 leader_election.cc:304] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e114f2373e14771981120579a59e39d, 4765facb107747888bcb47b17955a2b8; no voters:
I20260131 03:07:42.194885 22077 raft_consensus.cc:2804] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:07:42.195024 22077 raft_consensus.cc:697] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8 [term 1 LEADER]: Becoming Leader. State: Replica: 4765facb107747888bcb47b17955a2b8, State: Running, Role: LEADER
I20260131 03:07:42.195243 22077 consensus_queue.cc:237] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } } peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } }
I20260131 03:07:42.196110 21556 catalog_manager.cc:5654] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4765facb107747888bcb47b17955a2b8 (127.19.84.1). New cstate: current_term: 1 leader_uuid: "4765facb107747888bcb47b17955a2b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } health_report { overall_health: HEALTHY } } }
W20260131 03:07:42.202162 22048 tablet.cc:2378] T bbdd9c829e95461997a13db4067b58c5 P 7a3f75adeeeb4c008b60ba490d012604: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jan 31 03:07:42 dist-test-slave-5nnp krb5kdc[21509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828861, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:42.259231 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:42.252331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33359 (local address 127.0.0.1:54692)
0131 03:07:42.252537 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:42.252546 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:42.252611 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:42.252752 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:42.252752 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:42.253113 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:42.253278 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:42.253280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.254090 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:42.254091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:42.254528 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:42.254530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.254597 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:42.255130 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:42.255134 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:42.256691 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:42.258375 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:42.258381 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:42.258384 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:42.258598 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:42.258746 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:42.258746 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:42.258747 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:42.258780 (+ 33us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:42.258963 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:42.258971 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:42.259013 (+ 42us) client_negotiation.cc:769] Sending connection context
0131 03:07:42.259065 (+ 52us) client_negotiation.cc:240] Negotiation successful
0131 03:07:42.259155 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":107,"threads_started":1}
I20260131 03:07:42.259297 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:42.252417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54692 (local address 127.19.84.62:33359)
0131 03:07:42.252507 (+ 90us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:42.252510 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:42.252608 (+ 98us) server_negotiation.cc:408] Connection header received
0131 03:07:42.252658 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:42.252659 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:42.252669 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:42.252733 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:42.253383 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.253995 (+ 612us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:42.254645 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.254815 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:42.256828 (+ 2013us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:42.256834 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:42.256834 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:42.256840 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:42.258291 (+ 1451us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:42.258668 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:42.258669 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:42.258669 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:42.258690 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:42.258833 (+ 143us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:42.258833 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:42.258834 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:42.258916 (+ 82us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:42.258963 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:42.259096 (+ 133us) server_negotiation.cc:300] Negotiation successful
0131 03:07:42.259217 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":53}
I20260131 03:07:42.261351 21556 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:54692:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260131 03:07:42.261596 21556 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:07:42.263258 21981 tablet_service.cc:1505] Processing CreateTablet for tablet 32d4bebc88c74e8e92fd7c1429c3c4dd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3b34ee60a8844365b18bb2467fed8d21]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260131 03:07:42.263240 21839 tablet_service.cc:1505] Processing CreateTablet for tablet 32d4bebc88c74e8e92fd7c1429c3c4dd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3b34ee60a8844365b18bb2467fed8d21]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260131 03:07:42.263295 21696 tablet_service.cc:1505] Processing CreateTablet for tablet 32d4bebc88c74e8e92fd7c1429c3c4dd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3b34ee60a8844365b18bb2467fed8d21]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260131 03:07:42.263437 21839 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32d4bebc88c74e8e92fd7c1429c3c4dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:42.263437 21981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32d4bebc88c74e8e92fd7c1429c3c4dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:42.263491 21696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32d4bebc88c74e8e92fd7c1429c3c4dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:42.264595 22070 tablet_bootstrap.cc:492] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8: Bootstrap starting.
I20260131 03:07:42.265044 22072 tablet_bootstrap.cc:492] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 0e114f2373e14771981120579a59e39d: Bootstrap starting.
I20260131 03:07:42.265067 22070 tablet_bootstrap.cc:654] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:42.265588 22072 tablet_bootstrap.cc:654] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 0e114f2373e14771981120579a59e39d: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:42.265877 22070 tablet_bootstrap.cc:492] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8: No bootstrap required, opened a new log
I20260131 03:07:42.265949 22070 ts_tablet_manager.cc:1403] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:42.265952 22071 tablet_bootstrap.cc:492] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 7a3f75adeeeb4c008b60ba490d012604: Bootstrap starting.
I20260131 03:07:42.266132 22070 raft_consensus.cc:359] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:42.266223 22070 raft_consensus.cc:385] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:42.266249 22070 raft_consensus.cc:740] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4765facb107747888bcb47b17955a2b8, State: Initialized, Role: FOLLOWER
I20260131 03:07:42.266307 22070 consensus_queue.cc:260] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:42.266417 22071 tablet_bootstrap.cc:654] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 7a3f75adeeeb4c008b60ba490d012604: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:42.266428 22070 ts_tablet_manager.cc:1434] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260131 03:07:42.266484 22077 tablet_replica.cc:442] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } } }
I20260131 03:07:42.266589 22077 tablet_replica.cc:445] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:07:42.267058 22072 tablet_bootstrap.cc:492] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 0e114f2373e14771981120579a59e39d: No bootstrap required, opened a new log
I20260131 03:07:42.267117 22072 ts_tablet_manager.cc:1403] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 0e114f2373e14771981120579a59e39d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:07:42.267263 22072 raft_consensus.cc:359] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:42.267352 22072 raft_consensus.cc:385] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:42.267362 22071 tablet_bootstrap.cc:492] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 7a3f75adeeeb4c008b60ba490d012604: No bootstrap required, opened a new log
I20260131 03:07:42.267398 22072 raft_consensus.cc:740] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e114f2373e14771981120579a59e39d, State: Initialized, Role: FOLLOWER
I20260131 03:07:42.267438 22071 ts_tablet_manager.cc:1403] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 7a3f75adeeeb4c008b60ba490d012604: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:42.267449 22072 consensus_queue.cc:260] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 0e114f2373e14771981120579a59e39d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:42.267562 22072 ts_tablet_manager.cc:1434] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 0e114f2373e14771981120579a59e39d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260131 03:07:42.267586 22071 raft_consensus.cc:359] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:42.267671 22071 raft_consensus.cc:385] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:42.267704 22071 raft_consensus.cc:740] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a3f75adeeeb4c008b60ba490d012604, State: Initialized, Role: FOLLOWER
I20260131 03:07:42.267683 22078 tablet_replica.cc:442] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 0e114f2373e14771981120579a59e39d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } } }
I20260131 03:07:42.267745 22078 tablet_replica.cc:445] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 0e114f2373e14771981120579a59e39d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:07:42.267745 22071 consensus_queue.cc:260] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 7a3f75adeeeb4c008b60ba490d012604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:42.267867 22071 ts_tablet_manager.cc:1434] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 7a3f75adeeeb4c008b60ba490d012604: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:42.267928 22076 tablet_replica.cc:442] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 7a3f75adeeeb4c008b60ba490d012604: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } } }
I20260131 03:07:42.268021 22076 tablet_replica.cc:445] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 7a3f75adeeeb4c008b60ba490d012604: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:07:42.310739 22077 raft_consensus.cc:493] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:07:42.310858 22077 raft_consensus.cc:515] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:42.311034 22077 leader_election.cc:290] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e114f2373e14771981120579a59e39d (127.19.84.2:40287), 7a3f75adeeeb4c008b60ba490d012604 (127.19.84.3:38153)
I20260131 03:07:42.311226 21859 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32d4bebc88c74e8e92fd7c1429c3c4dd" candidate_uuid: "4765facb107747888bcb47b17955a2b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e114f2373e14771981120579a59e39d" is_pre_election: true
I20260131 03:07:42.311344 21859 raft_consensus.cc:2468] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4765facb107747888bcb47b17955a2b8 in term 0.
I20260131 03:07:42.311311 22001 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32d4bebc88c74e8e92fd7c1429c3c4dd" candidate_uuid: "4765facb107747888bcb47b17955a2b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a3f75adeeeb4c008b60ba490d012604" is_pre_election: true
I20260131 03:07:42.311425 22001 raft_consensus.cc:2468] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4765facb107747888bcb47b17955a2b8 in term 0.
I20260131 03:07:42.311558 21649 leader_election.cc:304] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e114f2373e14771981120579a59e39d, 4765facb107747888bcb47b17955a2b8; no voters:
I20260131 03:07:42.311673 22077 raft_consensus.cc:2804] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:07:42.311766 22077 raft_consensus.cc:493] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:07:42.311830 22077 raft_consensus.cc:3060] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:42.312460 22077 raft_consensus.cc:515] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:42.312633 22077 leader_election.cc:290] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8 [CANDIDATE]: Term 1 election: Requested vote from peers 0e114f2373e14771981120579a59e39d (127.19.84.2:40287), 7a3f75adeeeb4c008b60ba490d012604 (127.19.84.3:38153)
I20260131 03:07:42.312785 21859 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32d4bebc88c74e8e92fd7c1429c3c4dd" candidate_uuid: "4765facb107747888bcb47b17955a2b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e114f2373e14771981120579a59e39d"
I20260131 03:07:42.312868 22001 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32d4bebc88c74e8e92fd7c1429c3c4dd" candidate_uuid: "4765facb107747888bcb47b17955a2b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a3f75adeeeb4c008b60ba490d012604"
I20260131 03:07:42.312907 21859 raft_consensus.cc:3060] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:42.312950 22001 raft_consensus.cc:3060] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:42.313509 21859 raft_consensus.cc:2468] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 0e114f2373e14771981120579a59e39d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4765facb107747888bcb47b17955a2b8 in term 1.
I20260131 03:07:42.313513 22001 raft_consensus.cc:2468] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 7a3f75adeeeb4c008b60ba490d012604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4765facb107747888bcb47b17955a2b8 in term 1.
I20260131 03:07:42.313736 21647 leader_election.cc:304] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4765facb107747888bcb47b17955a2b8, 7a3f75adeeeb4c008b60ba490d012604; no voters:
I20260131 03:07:42.313890 22077 raft_consensus.cc:2804] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:07:42.313944 22077 raft_consensus.cc:697] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8 [term 1 LEADER]: Becoming Leader. State: Replica: 4765facb107747888bcb47b17955a2b8, State: Running, Role: LEADER
I20260131 03:07:42.314003 22077 consensus_queue.cc:237] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:42.314347 22081 tablet_replica.cc:442] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8: TxnStatusTablet state changed. Reason: New leader 4765facb107747888bcb47b17955a2b8. Latest consensus state: current_term: 1 leader_uuid: "4765facb107747888bcb47b17955a2b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } } }
I20260131 03:07:42.314430 22081 tablet_replica.cc:445] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8: This TxnStatusTablet replica's current role is: LEADER
I20260131 03:07:42.314615 22093 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260131 03:07:42.314728 22093 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260131 03:07:42.314677 21557 catalog_manager.cc:5654] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4765facb107747888bcb47b17955a2b8 (127.19.84.1). New cstate: current_term: 1 leader_uuid: "4765facb107747888bcb47b17955a2b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } health_report { overall_health: UNKNOWN } } }
I20260131 03:07:42.328787 22069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:42.326239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43054 (local address 127.19.84.1:46445)
0131 03:07:42.326302 (+ 63us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:42.326304 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:42.326315 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:07:42.326411 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:42.326412 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:42.326438 (+ 26us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:42.326508 (+ 70us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:07:42.326901 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.327496 (+ 595us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:42.328122 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.328366 (+ 244us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:42.328406 (+ 40us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:07:42.328507 (+ 101us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:07:42.328594 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:42.328676 (+ 82us) server_negotiation.cc:300] Negotiation successful
0131 03:07:42.328705 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260131 03:07:42.328827 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:42.326171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:46445 (local address 127.0.0.1:43054)
0131 03:07:42.326252 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:42.326267 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:42.326328 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:42.326517 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:42.326518 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:42.326520 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:07:42.326784 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:42.326788 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.327615 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:42.327616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:42.328018 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:42.328019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.328122 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:42.328144 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:07:42.328593 (+ 449us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:07:42.328607 (+ 14us) client_negotiation.cc:769] Sending connection context
0131 03:07:42.328673 (+ 66us) client_negotiation.cc:240] Negotiation successful
0131 03:07:42.328731 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260131 03:07:42.330677 22001 raft_consensus.cc:1275] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 7a3f75adeeeb4c008b60ba490d012604 [term 1 FOLLOWER]: Refusing update from remote peer 4765facb107747888bcb47b17955a2b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:07:42.330777 21859 raft_consensus.cc:1275] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 0e114f2373e14771981120579a59e39d [term 1 FOLLOWER]: Refusing update from remote peer 4765facb107747888bcb47b17955a2b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:07:42.331084 22081 consensus_queue.cc:1048] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:07:42.331192 22077 consensus_queue.cc:1048] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:07:42.332427 22078 tablet_replica.cc:442] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 0e114f2373e14771981120579a59e39d: TxnStatusTablet state changed. Reason: New leader 4765facb107747888bcb47b17955a2b8. Latest consensus state: current_term: 1 leader_uuid: "4765facb107747888bcb47b17955a2b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } } }
I20260131 03:07:42.332537 22078 tablet_replica.cc:445] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 0e114f2373e14771981120579a59e39d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:07:42.333598 22081 tablet_replica.cc:442] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4765facb107747888bcb47b17955a2b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } } }
I20260131 03:07:42.333690 22081 tablet_replica.cc:445] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8: This TxnStatusTablet replica's current role is: LEADER
I20260131 03:07:42.333779 22076 tablet_replica.cc:442] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 7a3f75adeeeb4c008b60ba490d012604: TxnStatusTablet state changed. Reason: New leader 4765facb107747888bcb47b17955a2b8. Latest consensus state: current_term: 1 leader_uuid: "4765facb107747888bcb47b17955a2b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } } }
I20260131 03:07:42.333866 22076 tablet_replica.cc:445] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 7a3f75adeeeb4c008b60ba490d012604: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:07:42.334280 22078 tablet_replica.cc:442] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 0e114f2373e14771981120579a59e39d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4765facb107747888bcb47b17955a2b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } } }
I20260131 03:07:42.334358 22078 tablet_replica.cc:445] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 0e114f2373e14771981120579a59e39d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:07:42.334558 22097 mvcc.cc:204] Tried to move back new op lower bound from 7249219020103524352 to 7249219020038553600. Current Snapshot: MvccSnapshot[applied={T|T < 7249219020103524352 or (T in {7249219020103524352})}]
I20260131 03:07:42.336028 22077 tablet_replica.cc:442] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4765facb107747888bcb47b17955a2b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } } }
I20260131 03:07:42.336114 22077 tablet_replica.cc:445] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 4765facb107747888bcb47b17955a2b8: This TxnStatusTablet replica's current role is: LEADER
I20260131 03:07:42.336927 22076 tablet_replica.cc:442] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 7a3f75adeeeb4c008b60ba490d012604: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4765facb107747888bcb47b17955a2b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } } }
I20260131 03:07:42.337016 22076 tablet_replica.cc:445] T 32d4bebc88c74e8e92fd7c1429c3c4dd P 7a3f75adeeeb4c008b60ba490d012604: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:07:42.337284 22099 mvcc.cc:204] Tried to move back new op lower bound from 7249219020103524352 to 7249219020038553600. Current Snapshot: MvccSnapshot[applied={T|T < 7249219020103524352 or (T in {7249219020103524352})}]
I20260131 03:07:42.339777 22069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:42.337761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43060 (local address 127.19.84.1:46445)
0131 03:07:42.337860 (+ 99us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:42.337861 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:42.337867 (+ 6us) server_negotiation.cc:408] Connection header received
0131 03:07:42.337886 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:42.337886 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:42.337894 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:42.337957 (+ 63us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:07:42.338193 (+ 236us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.338648 (+ 455us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:42.339236 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.339452 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:42.339472 (+ 20us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:07:42.339497 (+ 25us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:07:42.339563 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:42.339675 (+ 112us) server_negotiation.cc:300] Negotiation successful
0131 03:07:42.339699 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260131 03:07:42.349689 22069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:42.342424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:44527 (local address 127.19.84.1:46445)
0131 03:07:42.342474 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:42.342476 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:42.347668 (+ 5192us) server_negotiation.cc:408] Connection header received
0131 03:07:42.347766 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:42.347767 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:42.347777 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:42.347872 (+ 95us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:07:42.348155 (+ 283us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.348632 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:42.349241 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.349385 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:42.349401 (+ 16us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:07:42.349422 (+ 21us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:07:42.349483 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:42.349573 (+ 90us) server_negotiation.cc:300] Negotiation successful
0131 03:07:42.349593 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260131 03:07:42.349701 22107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:42.342356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:46445 (local address 127.19.84.1:44527)
0131 03:07:42.347577 (+ 5221us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:42.347590 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:42.347688 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:42.347876 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:42.347878 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:42.347879 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:07:42.348083 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:42.348084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.348717 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:42.348718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:42.349149 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:42.349151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.349232 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:42.349253 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:07:42.349500 (+ 247us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:07:42.349502 (+ 2us) client_negotiation.cc:769] Sending connection context
0131 03:07:42.349565 (+ 63us) client_negotiation.cc:240] Negotiation successful
0131 03:07:42.349608 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5172,"thread_start_us":90,"threads_started":1}
I20260131 03:07:42.352298 22001 raft_consensus.cc:1275] T bbdd9c829e95461997a13db4067b58c5 P 7a3f75adeeeb4c008b60ba490d012604 [term 1 FOLLOWER]: Refusing update from remote peer 4765facb107747888bcb47b17955a2b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:07:42.352305 21859 raft_consensus.cc:1275] T bbdd9c829e95461997a13db4067b58c5 P 0e114f2373e14771981120579a59e39d [term 1 FOLLOWER]: Refusing update from remote peer 4765facb107747888bcb47b17955a2b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:07:42.352533 22077 consensus_queue.cc:1048] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:07:42.352705 22108 consensus_queue.cc:1048] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:07:42.365458 21556 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:07:42.365634 21556 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:07:42.366369 21556 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537: Sending DeleteTablet for 3 replicas of tablet bbdd9c829e95461997a13db4067b58c5
I20260131 03:07:42.366703 21696 tablet_service.cc:1552] Processing DeleteTablet for tablet bbdd9c829e95461997a13db4067b58c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:07:42 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:43038
I20260131 03:07:42.366714 21981 tablet_service.cc:1552] Processing DeleteTablet for tablet bbdd9c829e95461997a13db4067b58c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:07:42 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:46458
I20260131 03:07:42.366973 21839 tablet_service.cc:1552] Processing DeleteTablet for tablet bbdd9c829e95461997a13db4067b58c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:07:42 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:38498
I20260131 03:07:42.367372 22113 tablet_replica.cc:333] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8: stopping tablet replica
I20260131 03:07:42.367503 22113 raft_consensus.cc:2243] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:07:42.367466 21556 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:07:42.367697 21556 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:07:42.367496 22115 tablet_replica.cc:333] T bbdd9c829e95461997a13db4067b58c5 P 0e114f2373e14771981120579a59e39d: stopping tablet replica
I20260131 03:07:42.367707 22113 raft_consensus.cc:2272] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:07:42.367985 22115 raft_consensus.cc:2243] T bbdd9c829e95461997a13db4067b58c5 P 0e114f2373e14771981120579a59e39d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:07:42.368063 22115 raft_consensus.cc:2272] T bbdd9c829e95461997a13db4067b58c5 P 0e114f2373e14771981120579a59e39d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:07:42.368510 22115 ts_tablet_manager.cc:1916] T bbdd9c829e95461997a13db4067b58c5 P 0e114f2373e14771981120579a59e39d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:07:42.369441 21696 tablet_service.cc:1505] Processing CreateTablet for tablet dfa7a90cd6534b8ebd190270df300edd (DEFAULT_TABLE table=test-table [id=1ba3ef635ef645209aad70724b46d4f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:07:42.369468 21981 tablet_service.cc:1505] Processing CreateTablet for tablet dfa7a90cd6534b8ebd190270df300edd (DEFAULT_TABLE table=test-table [id=1ba3ef635ef645209aad70724b46d4f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:07:42.369561 21696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa7a90cd6534b8ebd190270df300edd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:42.369561 21981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa7a90cd6534b8ebd190270df300edd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:42.370743 22115 ts_tablet_manager.cc:1929] T bbdd9c829e95461997a13db4067b58c5 P 0e114f2373e14771981120579a59e39d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:07:42.370810 22115 log.cc:1199] T bbdd9c829e95461997a13db4067b58c5 P 0e114f2373e14771981120579a59e39d: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-1/wal/wals/bbdd9c829e95461997a13db4067b58c5
I20260131 03:07:42.370851 22070 tablet_bootstrap.cc:492] T dfa7a90cd6534b8ebd190270df300edd P 4765facb107747888bcb47b17955a2b8: Bootstrap starting.
I20260131 03:07:42.370848 22071 tablet_bootstrap.cc:492] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604: Bootstrap starting.
I20260131 03:07:42.371086 22115 ts_tablet_manager.cc:1950] T bbdd9c829e95461997a13db4067b58c5 P 0e114f2373e14771981120579a59e39d: Deleting consensus metadata
I20260131 03:07:42.371318 22070 tablet_bootstrap.cc:654] T dfa7a90cd6534b8ebd190270df300edd P 4765facb107747888bcb47b17955a2b8: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:42.371318 22071 tablet_bootstrap.cc:654] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:42.371927 21543 catalog_manager.cc:4985] TS 0e114f2373e14771981120579a59e39d (127.19.84.2:40287): tablet bbdd9c829e95461997a13db4067b58c5 (table test-table [id=4ff6ab6f622f42e4ab3ce5ed065411b4]) successfully deleted
I20260131 03:07:42.372082 22070 tablet_bootstrap.cc:492] T dfa7a90cd6534b8ebd190270df300edd P 4765facb107747888bcb47b17955a2b8: No bootstrap required, opened a new log
I20260131 03:07:42.372102 21839 tablet_service.cc:1505] Processing CreateTablet for tablet dfa7a90cd6534b8ebd190270df300edd (DEFAULT_TABLE table=test-table [id=1ba3ef635ef645209aad70724b46d4f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:07:42.372128 22070 ts_tablet_manager.cc:1403] T dfa7a90cd6534b8ebd190270df300edd P 4765facb107747888bcb47b17955a2b8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260131 03:07:42.372224 21839 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa7a90cd6534b8ebd190270df300edd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:42.372304 22070 raft_consensus.cc:359] T dfa7a90cd6534b8ebd190270df300edd P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:42.372380 22070 raft_consensus.cc:385] T dfa7a90cd6534b8ebd190270df300edd P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:42.372401 22070 raft_consensus.cc:740] T dfa7a90cd6534b8ebd190270df300edd P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4765facb107747888bcb47b17955a2b8, State: Initialized, Role: FOLLOWER
I20260131 03:07:42.372442 22070 consensus_queue.cc:260] T dfa7a90cd6534b8ebd190270df300edd P 4765facb107747888bcb47b17955a2b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:42.372524 22070 ts_tablet_manager.cc:1434] T dfa7a90cd6534b8ebd190270df300edd P 4765facb107747888bcb47b17955a2b8: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260131 03:07:42.376565 22072 tablet_bootstrap.cc:492] T dfa7a90cd6534b8ebd190270df300edd P 0e114f2373e14771981120579a59e39d: Bootstrap starting.
I20260131 03:07:42.376976 22072 tablet_bootstrap.cc:654] T dfa7a90cd6534b8ebd190270df300edd P 0e114f2373e14771981120579a59e39d: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:42.368674 22113 ts_tablet_manager.cc:1916] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:07:42.377701 22072 tablet_bootstrap.cc:492] T dfa7a90cd6534b8ebd190270df300edd P 0e114f2373e14771981120579a59e39d: No bootstrap required, opened a new log
I20260131 03:07:42.377751 22072 ts_tablet_manager.cc:1403] T dfa7a90cd6534b8ebd190270df300edd P 0e114f2373e14771981120579a59e39d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:42.377934 22072 raft_consensus.cc:359] T dfa7a90cd6534b8ebd190270df300edd P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:42.378002 22072 raft_consensus.cc:385] T dfa7a90cd6534b8ebd190270df300edd P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:42.378026 22072 raft_consensus.cc:740] T dfa7a90cd6534b8ebd190270df300edd P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e114f2373e14771981120579a59e39d, State: Initialized, Role: FOLLOWER
I20260131 03:07:42.378078 22072 consensus_queue.cc:260] T dfa7a90cd6534b8ebd190270df300edd P 0e114f2373e14771981120579a59e39d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:42.378165 22072 ts_tablet_manager.cc:1434] T dfa7a90cd6534b8ebd190270df300edd P 0e114f2373e14771981120579a59e39d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:42.379679 22113 ts_tablet_manager.cc:1929] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:07:42.379778 22113 log.cc:1199] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-0/wal/wals/bbdd9c829e95461997a13db4067b58c5
I20260131 03:07:42.380023 22113 ts_tablet_manager.cc:1950] T bbdd9c829e95461997a13db4067b58c5 P 4765facb107747888bcb47b17955a2b8: Deleting consensus metadata
I20260131 03:07:42.380275 22114 tablet_replica.cc:333] T bbdd9c829e95461997a13db4067b58c5 P 7a3f75adeeeb4c008b60ba490d012604: stopping tablet replica
I20260131 03:07:42.380389 22114 raft_consensus.cc:2243] T bbdd9c829e95461997a13db4067b58c5 P 7a3f75adeeeb4c008b60ba490d012604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:07:42.380457 22114 raft_consensus.cc:2272] T bbdd9c829e95461997a13db4067b58c5 P 7a3f75adeeeb4c008b60ba490d012604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:07:42.380990 21544 catalog_manager.cc:4985] TS 4765facb107747888bcb47b17955a2b8 (127.19.84.1:46445): tablet bbdd9c829e95461997a13db4067b58c5 (table test-table [id=4ff6ab6f622f42e4ab3ce5ed065411b4]) successfully deleted
I20260131 03:07:42.381354 22071 tablet_bootstrap.cc:492] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604: No bootstrap required, opened a new log
I20260131 03:07:42.381428 22071 ts_tablet_manager.cc:1403] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.001s
I20260131 03:07:42.381640 22071 raft_consensus.cc:359] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:42.381726 22071 raft_consensus.cc:385] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:42.381754 22071 raft_consensus.cc:740] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a3f75adeeeb4c008b60ba490d012604, State: Initialized, Role: FOLLOWER
I20260131 03:07:42.381798 22071 consensus_queue.cc:260] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:42.381894 22071 ts_tablet_manager.cc:1434] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:42.384049 22114 ts_tablet_manager.cc:1916] T bbdd9c829e95461997a13db4067b58c5 P 7a3f75adeeeb4c008b60ba490d012604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:07:42.408499 22114 ts_tablet_manager.cc:1929] T bbdd9c829e95461997a13db4067b58c5 P 7a3f75adeeeb4c008b60ba490d012604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:07:42.408631 22114 log.cc:1199] T bbdd9c829e95461997a13db4067b58c5 P 7a3f75adeeeb4c008b60ba490d012604: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/bbdd9c829e95461997a13db4067b58c5
I20260131 03:07:42.408917 22114 ts_tablet_manager.cc:1950] T bbdd9c829e95461997a13db4067b58c5 P 7a3f75adeeeb4c008b60ba490d012604: Deleting consensus metadata
I20260131 03:07:42.409405 21541 catalog_manager.cc:4985] TS 7a3f75adeeeb4c008b60ba490d012604 (127.19.84.3:38153): tablet bbdd9c829e95461997a13db4067b58c5 (table test-table [id=4ff6ab6f622f42e4ab3ce5ed065411b4]) successfully deleted
I20260131 03:07:42.430155 22076 raft_consensus.cc:493] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:07:42.430266 22076 raft_consensus.cc:515] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:42.430594 22076 leader_election.cc:290] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4765facb107747888bcb47b17955a2b8 (127.19.84.1:46445), 0e114f2373e14771981120579a59e39d (127.19.84.2:40287)
I20260131 03:07:42.433041 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:42.430785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:46445 (local address 127.19.84.3:54119)
0131 03:07:42.430841 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:42.430855 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:42.430898 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:42.431095 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:42.431096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:42.431099 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:07:42.431313 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:42.431316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.432048 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:42.432049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:42.432818 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:42.432820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.432893 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:42.432898 (+ 5us) client_negotiation.cc:769] Sending connection context
0131 03:07:42.432931 (+ 33us) client_negotiation.cc:240] Negotiation successful
0131 03:07:42.432969 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260131 03:07:42.433483 22069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:42.430858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:54119 (local address 127.19.84.1:46445)
0131 03:07:42.430902 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:42.430904 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:42.430943 (+ 39us) server_negotiation.cc:408] Connection header received
0131 03:07:42.430970 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:42.430971 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:42.430981 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:42.431031 (+ 50us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:07:42.431430 (+ 399us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.431965 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:42.432915 (+ 950us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.433355 (+ 440us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:42.433365 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:42.433390 (+ 25us) server_negotiation.cc:300] Negotiation successful
0131 03:07:42.433422 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260131 03:07:42.433739 21716 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfa7a90cd6534b8ebd190270df300edd" candidate_uuid: "7a3f75adeeeb4c008b60ba490d012604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4765facb107747888bcb47b17955a2b8" is_pre_election: true
I20260131 03:07:42.433854 21716 raft_consensus.cc:2468] T dfa7a90cd6534b8ebd190270df300edd P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a3f75adeeeb4c008b60ba490d012604 in term 0.
I20260131 03:07:42.434011 22067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:42.430984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:40467 (local address 127.19.84.2:40287)
0131 03:07:42.431027 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:42.431029 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:42.431063 (+ 34us) server_negotiation.cc:408] Connection header received
0131 03:07:42.431165 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:42.431171 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:42.431180 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:42.431245 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:07:42.431652 (+ 407us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.432336 (+ 684us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:42.433342 (+ 1006us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.433762 (+ 420us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:42.433768 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:42.433907 (+ 139us) server_negotiation.cc:300] Negotiation successful
0131 03:07:42.433944 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260131 03:07:42.434039 21935 leader_election.cc:304] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4765facb107747888bcb47b17955a2b8, 7a3f75adeeeb4c008b60ba490d012604; no voters:
I20260131 03:07:42.434180 22076 raft_consensus.cc:2804] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:07:42.434252 22076 raft_consensus.cc:493] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:07:42.434281 22076 raft_consensus.cc:3060] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:42.434360 22119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:42.430786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:40287 (local address 127.19.84.3:40467)
0131 03:07:42.430998 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:42.431008 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:42.431086 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:42.431248 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:42.431249 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:42.431251 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:07:42.431545 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:42.431548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.432433 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:42.432434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:42.433256 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:42.433259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.433794 (+ 535us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:42.433796 (+ 2us) client_negotiation.cc:769] Sending connection context
0131 03:07:42.434128 (+ 332us) client_negotiation.cc:240] Negotiation successful
0131 03:07:42.434171 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":12,"thread_start_us":85,"threads_started":1}
I20260131 03:07:42.434615 21859 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfa7a90cd6534b8ebd190270df300edd" candidate_uuid: "7a3f75adeeeb4c008b60ba490d012604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e114f2373e14771981120579a59e39d" is_pre_election: true
I20260131 03:07:42.434692 21859 raft_consensus.cc:2468] T dfa7a90cd6534b8ebd190270df300edd P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a3f75adeeeb4c008b60ba490d012604 in term 0.
I20260131 03:07:42.435161 22076 raft_consensus.cc:515] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:42.435297 22076 leader_election.cc:290] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604 [CANDIDATE]: Term 1 election: Requested vote from peers 4765facb107747888bcb47b17955a2b8 (127.19.84.1:46445), 0e114f2373e14771981120579a59e39d (127.19.84.2:40287)
I20260131 03:07:42.435470 21859 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfa7a90cd6534b8ebd190270df300edd" candidate_uuid: "7a3f75adeeeb4c008b60ba490d012604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e114f2373e14771981120579a59e39d"
I20260131 03:07:42.435488 21716 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfa7a90cd6534b8ebd190270df300edd" candidate_uuid: "7a3f75adeeeb4c008b60ba490d012604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4765facb107747888bcb47b17955a2b8"
I20260131 03:07:42.435549 21859 raft_consensus.cc:3060] T dfa7a90cd6534b8ebd190270df300edd P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:42.435563 21716 raft_consensus.cc:3060] T dfa7a90cd6534b8ebd190270df300edd P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:42.436143 21859 raft_consensus.cc:2468] T dfa7a90cd6534b8ebd190270df300edd P 0e114f2373e14771981120579a59e39d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a3f75adeeeb4c008b60ba490d012604 in term 1.
I20260131 03:07:42.436340 21934 leader_election.cc:304] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e114f2373e14771981120579a59e39d, 7a3f75adeeeb4c008b60ba490d012604; no voters:
I20260131 03:07:42.436401 21716 raft_consensus.cc:2468] T dfa7a90cd6534b8ebd190270df300edd P 4765facb107747888bcb47b17955a2b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a3f75adeeeb4c008b60ba490d012604 in term 1.
I20260131 03:07:42.436524 22076 raft_consensus.cc:2804] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:07:42.436640 22076 raft_consensus.cc:697] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604 [term 1 LEADER]: Becoming Leader. State: Replica: 7a3f75adeeeb4c008b60ba490d012604, State: Running, Role: LEADER
I20260131 03:07:42.436715 22076 consensus_queue.cc:237] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:42.437358 21556 catalog_manager.cc:5654] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a3f75adeeeb4c008b60ba490d012604 (127.19.84.3). New cstate: current_term: 1 leader_uuid: "7a3f75adeeeb4c008b60ba490d012604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } health_report { overall_health: HEALTHY } } }
I20260131 03:07:42.447252 22068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:42.444923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46460 (local address 127.19.84.3:38153)
0131 03:07:42.444973 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:42.444975 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:42.444984 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:07:42.445041 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:42.445042 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:42.445056 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:42.445100 (+ 44us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:07:42.445405 (+ 305us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.445884 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:42.446740 (+ 856us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.446876 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:42.446904 (+ 28us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:07:42.446989 (+ 85us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:07:42.447048 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:42.447137 (+ 89us) server_negotiation.cc:300] Negotiation successful
0131 03:07:42.447157 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260131 03:07:42.452126 22053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:42.449603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:46445 (local address 127.19.84.3:55683)
0131 03:07:42.449730 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:42.449739 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:42.449778 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:42.449978 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:42.449979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:42.449985 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:07:42.450152 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:42.450168 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.451137 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:42.451139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:42.451603 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:42.451606 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.451667 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:42.451679 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:07:42.451978 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:07:42.451979 (+ 1us) client_negotiation.cc:769] Sending connection context
0131 03:07:42.452022 (+ 43us) client_negotiation.cc:240] Negotiation successful
0131 03:07:42.452058 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260131 03:07:42.452140 22069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:42.449672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:55683 (local address 127.19.84.1:46445)
0131 03:07:42.449738 (+ 66us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:42.449740 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:42.449770 (+ 30us) server_negotiation.cc:408] Connection header received
0131 03:07:42.449834 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:42.449835 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:42.449845 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:42.449911 (+ 66us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:07:42.450264 (+ 353us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.450979 (+ 715us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:42.451686 (+ 707us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.451852 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:42.451875 (+ 23us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:07:42.451901 (+ 26us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:07:42.451965 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:42.452048 (+ 83us) server_negotiation.cc:300] Negotiation successful
0131 03:07:42.452073 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260131 03:07:42.454463 21859 raft_consensus.cc:1275] T dfa7a90cd6534b8ebd190270df300edd P 0e114f2373e14771981120579a59e39d [term 1 FOLLOWER]: Refusing update from remote peer 7a3f75adeeeb4c008b60ba490d012604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:07:42.454578 21716 raft_consensus.cc:1275] T dfa7a90cd6534b8ebd190270df300edd P 4765facb107747888bcb47b17955a2b8 [term 1 FOLLOWER]: Refusing update from remote peer 7a3f75adeeeb4c008b60ba490d012604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:07:42.454699 22120 consensus_queue.cc:1048] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:07:42.454795 22076 consensus_queue.cc:1048] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:07:42.462508 22107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:42.460097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:38153 (local address 127.19.84.1:57011)
0131 03:07:42.460139 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:42.460150 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:42.460185 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:42.460368 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:42.460370 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:42.460371 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:07:42.460565 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:42.460567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.461374 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:42.461375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:42.461953 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:42.461956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.462032 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:42.462036 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:07:42.462338 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:07:42.462340 (+ 2us) client_negotiation.cc:769] Sending connection context
0131 03:07:42.462391 (+ 51us) client_negotiation.cc:240] Negotiation successful
0131 03:07:42.462437 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260131 03:07:42.462770 22068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:42.460218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:57011 (local address 127.19.84.3:38153)
0131 03:07:42.460273 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:42.460275 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:42.460282 (+ 7us) server_negotiation.cc:408] Connection header received
0131 03:07:42.460302 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:42.460303 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:42.460313 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:42.460364 (+ 51us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:07:42.460639 (+ 275us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.461300 (+ 661us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:42.462078 (+ 778us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:42.462195 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:42.462213 (+ 18us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:07:42.462267 (+ 54us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:07:42.462325 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:42.462686 (+ 361us) server_negotiation.cc:300] Negotiation successful
0131 03:07:42.462710 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260131 03:07:42.463222 22099 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20260131 03:07:43.984959 21557 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:07:43.985070 21557 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:07:43.986016 21557 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537: Sending DeleteTablet for 3 replicas of tablet dfa7a90cd6534b8ebd190270df300edd
I20260131 03:07:43.986306 21695 tablet_service.cc:1552] Processing DeleteTablet for tablet dfa7a90cd6534b8ebd190270df300edd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:07:43 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:43038
I20260131 03:07:43.986327 21981 tablet_service.cc:1552] Processing DeleteTablet for tablet dfa7a90cd6534b8ebd190270df300edd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:07:43 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:46458
I20260131 03:07:43.986370 21839 tablet_service.cc:1552] Processing DeleteTablet for tablet dfa7a90cd6534b8ebd190270df300edd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:07:43 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:38498
I20260131 03:07:43.986577 22148 tablet_replica.cc:333] T dfa7a90cd6534b8ebd190270df300edd P 4765facb107747888bcb47b17955a2b8: stopping tablet replica
I20260131 03:07:43.986575 22150 tablet_replica.cc:333] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604: stopping tablet replica
I20260131 03:07:43.986680 22148 raft_consensus.cc:2243] T dfa7a90cd6534b8ebd190270df300edd P 4765facb107747888bcb47b17955a2b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:07:43.986739 22148 raft_consensus.cc:2272] T dfa7a90cd6534b8ebd190270df300edd P 4765facb107747888bcb47b17955a2b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:07:43.986963 22150 raft_consensus.cc:2243] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:07:43.987027 22148 ts_tablet_manager.cc:1916] T dfa7a90cd6534b8ebd190270df300edd P 4765facb107747888bcb47b17955a2b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:07:43.987089 22150 raft_consensus.cc:2272] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:07:43.987341 22150 ts_tablet_manager.cc:1916] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:07:43.988934 22148 ts_tablet_manager.cc:1929] T dfa7a90cd6534b8ebd190270df300edd P 4765facb107747888bcb47b17955a2b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260131 03:07:43.989010 22148 log.cc:1199] T dfa7a90cd6534b8ebd190270df300edd P 4765facb107747888bcb47b17955a2b8: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-0/wal/wals/dfa7a90cd6534b8ebd190270df300edd
I20260131 03:07:43.989009 22150 ts_tablet_manager.cc:1929] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260131 03:07:43.989056 22150 log.cc:1199] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/dfa7a90cd6534b8ebd190270df300edd
I20260131 03:07:43.989290 22148 ts_tablet_manager.cc:1950] T dfa7a90cd6534b8ebd190270df300edd P 4765facb107747888bcb47b17955a2b8: Deleting consensus metadata
I20260131 03:07:43.989290 22150 ts_tablet_manager.cc:1950] T dfa7a90cd6534b8ebd190270df300edd P 7a3f75adeeeb4c008b60ba490d012604: Deleting consensus metadata
I20260131 03:07:43.989599 21541 catalog_manager.cc:4985] TS 7a3f75adeeeb4c008b60ba490d012604 (127.19.84.3:38153): tablet dfa7a90cd6534b8ebd190270df300edd (table test-table [id=1ba3ef635ef645209aad70724b46d4f4]) successfully deleted
I20260131 03:07:43.989640 21544 catalog_manager.cc:4985] TS 4765facb107747888bcb47b17955a2b8 (127.19.84.1:46445): tablet dfa7a90cd6534b8ebd190270df300edd (table test-table [id=1ba3ef635ef645209aad70724b46d4f4]) successfully deleted
I20260131 03:07:43.991515 22149 tablet_replica.cc:333] T dfa7a90cd6534b8ebd190270df300edd P 0e114f2373e14771981120579a59e39d: stopping tablet replica
I20260131 03:07:43.991593 22149 raft_consensus.cc:2243] T dfa7a90cd6534b8ebd190270df300edd P 0e114f2373e14771981120579a59e39d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:07:43.991657 22149 raft_consensus.cc:2272] T dfa7a90cd6534b8ebd190270df300edd P 0e114f2373e14771981120579a59e39d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:07:43.991917 22149 ts_tablet_manager.cc:1916] T dfa7a90cd6534b8ebd190270df300edd P 0e114f2373e14771981120579a59e39d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:07:43.993386 22149 ts_tablet_manager.cc:1929] T dfa7a90cd6534b8ebd190270df300edd P 0e114f2373e14771981120579a59e39d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260131 03:07:43.993467 22149 log.cc:1199] T dfa7a90cd6534b8ebd190270df300edd P 0e114f2373e14771981120579a59e39d: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-1/wal/wals/dfa7a90cd6534b8ebd190270df300edd
I20260131 03:07:43.993686 22149 ts_tablet_manager.cc:1950] T dfa7a90cd6534b8ebd190270df300edd P 0e114f2373e14771981120579a59e39d: Deleting consensus metadata
I20260131 03:07:43.994064 21543 catalog_manager.cc:4985] TS 0e114f2373e14771981120579a59e39d (127.19.84.2:40287): tablet dfa7a90cd6534b8ebd190270df300edd (table test-table [id=1ba3ef635ef645209aad70724b46d4f4]) successfully deleted
Jan 31 03:07:43 dist-test-slave-5nnp krb5kdc[21509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828863, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 31 03:07:44 dist-test-slave-5nnp krb5kdc[21509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828863, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:44.010838 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:44.003798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54708 (local address 127.19.84.62:33359)
0131 03:07:44.003954 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:44.003958 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:44.004038 (+ 80us) server_negotiation.cc:408] Connection header received
0131 03:07:44.004143 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:44.004145 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:44.004157 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:44.004242 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:44.004952 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:44.005475 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:44.006065 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:44.006258 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:44.008357 (+ 2099us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:44.008362 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:44.008363 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:44.008370 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:44.009686 (+ 1316us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:44.010039 (+ 353us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:44.010041 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:44.010041 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:44.010071 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:44.010239 (+ 168us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:44.010240 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:44.010240 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:44.010345 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:44.010462 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:44.010583 (+ 121us) server_negotiation.cc:300] Negotiation successful
0131 03:07:44.010712 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
I20260131 03:07:44.012523 21557 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:07:44.012679 21557 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:07:44.014304 21695 tablet_service.cc:1505] Processing CreateTablet for tablet 71d25857034c4f9c8c90119d5a3ac48b (DEFAULT_TABLE table=test-table [id=07143169f360403ba5e600bdba2c10ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:07:44.014330 21981 tablet_service.cc:1505] Processing CreateTablet for tablet 71d25857034c4f9c8c90119d5a3ac48b (DEFAULT_TABLE table=test-table [id=07143169f360403ba5e600bdba2c10ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:07:44.014325 21839 tablet_service.cc:1505] Processing CreateTablet for tablet 71d25857034c4f9c8c90119d5a3ac48b (DEFAULT_TABLE table=test-table [id=07143169f360403ba5e600bdba2c10ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:07:44.014463 21981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71d25857034c4f9c8c90119d5a3ac48b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:44.014474 21839 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71d25857034c4f9c8c90119d5a3ac48b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:44.014465 21695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71d25857034c4f9c8c90119d5a3ac48b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:44.015676 22159 tablet_bootstrap.cc:492] T 71d25857034c4f9c8c90119d5a3ac48b P 0e114f2373e14771981120579a59e39d: Bootstrap starting.
I20260131 03:07:44.016083 22159 tablet_bootstrap.cc:654] T 71d25857034c4f9c8c90119d5a3ac48b P 0e114f2373e14771981120579a59e39d: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:44.016281 22160 tablet_bootstrap.cc:492] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604: Bootstrap starting.
I20260131 03:07:44.016593 22161 tablet_bootstrap.cc:492] T 71d25857034c4f9c8c90119d5a3ac48b P 4765facb107747888bcb47b17955a2b8: Bootstrap starting.
I20260131 03:07:44.016685 22160 tablet_bootstrap.cc:654] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:44.017000 22159 tablet_bootstrap.cc:492] T 71d25857034c4f9c8c90119d5a3ac48b P 0e114f2373e14771981120579a59e39d: No bootstrap required, opened a new log
I20260131 03:07:44.017078 22159 ts_tablet_manager.cc:1403] T 71d25857034c4f9c8c90119d5a3ac48b P 0e114f2373e14771981120579a59e39d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:44.017098 22161 tablet_bootstrap.cc:654] T 71d25857034c4f9c8c90119d5a3ac48b P 4765facb107747888bcb47b17955a2b8: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:44.017266 22159 raft_consensus.cc:359] T 71d25857034c4f9c8c90119d5a3ac48b P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:44.017356 22160 tablet_bootstrap.cc:492] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604: No bootstrap required, opened a new log
I20260131 03:07:44.017421 22160 ts_tablet_manager.cc:1403] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:44.017436 22159 raft_consensus.cc:385] T 71d25857034c4f9c8c90119d5a3ac48b P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:44.017465 22159 raft_consensus.cc:740] T 71d25857034c4f9c8c90119d5a3ac48b P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e114f2373e14771981120579a59e39d, State: Initialized, Role: FOLLOWER
I20260131 03:07:44.017562 22159 consensus_queue.cc:260] T 71d25857034c4f9c8c90119d5a3ac48b P 0e114f2373e14771981120579a59e39d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:44.017812 22159 ts_tablet_manager.cc:1434] T 71d25857034c4f9c8c90119d5a3ac48b P 0e114f2373e14771981120579a59e39d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:44.017827 22161 tablet_bootstrap.cc:492] T 71d25857034c4f9c8c90119d5a3ac48b P 4765facb107747888bcb47b17955a2b8: No bootstrap required, opened a new log
I20260131 03:07:44.017802 22160 raft_consensus.cc:359] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:44.017905 22161 ts_tablet_manager.cc:1403] T 71d25857034c4f9c8c90119d5a3ac48b P 4765facb107747888bcb47b17955a2b8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:44.017954 22160 raft_consensus.cc:385] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:44.017997 22160 raft_consensus.cc:740] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a3f75adeeeb4c008b60ba490d012604, State: Initialized, Role: FOLLOWER
I20260131 03:07:44.018052 22160 consensus_queue.cc:260] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:44.018154 22160 ts_tablet_manager.cc:1434] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260131 03:07:44.018151 22161 raft_consensus.cc:359] T 71d25857034c4f9c8c90119d5a3ac48b P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:44.018260 22161 raft_consensus.cc:385] T 71d25857034c4f9c8c90119d5a3ac48b P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:44.018298 22161 raft_consensus.cc:740] T 71d25857034c4f9c8c90119d5a3ac48b P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4765facb107747888bcb47b17955a2b8, State: Initialized, Role: FOLLOWER
I20260131 03:07:44.018348 22161 consensus_queue.cc:260] T 71d25857034c4f9c8c90119d5a3ac48b P 4765facb107747888bcb47b17955a2b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:44.018497 22161 ts_tablet_manager.cc:1434] T 71d25857034c4f9c8c90119d5a3ac48b P 4765facb107747888bcb47b17955a2b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260131 03:07:44.369973 22076 raft_consensus.cc:493] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:07:44.370129 22076 raft_consensus.cc:515] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:44.370291 22076 leader_election.cc:290] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4765facb107747888bcb47b17955a2b8 (127.19.84.1:46445), 0e114f2373e14771981120579a59e39d (127.19.84.2:40287)
I20260131 03:07:44.370518 21859 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71d25857034c4f9c8c90119d5a3ac48b" candidate_uuid: "7a3f75adeeeb4c008b60ba490d012604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e114f2373e14771981120579a59e39d" is_pre_election: true
I20260131 03:07:44.370558 21716 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71d25857034c4f9c8c90119d5a3ac48b" candidate_uuid: "7a3f75adeeeb4c008b60ba490d012604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4765facb107747888bcb47b17955a2b8" is_pre_election: true
I20260131 03:07:44.370633 21716 raft_consensus.cc:2468] T 71d25857034c4f9c8c90119d5a3ac48b P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a3f75adeeeb4c008b60ba490d012604 in term 0.
I20260131 03:07:44.370652 21859 raft_consensus.cc:2468] T 71d25857034c4f9c8c90119d5a3ac48b P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a3f75adeeeb4c008b60ba490d012604 in term 0.
I20260131 03:07:44.370779 21935 leader_election.cc:304] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4765facb107747888bcb47b17955a2b8, 7a3f75adeeeb4c008b60ba490d012604; no voters:
I20260131 03:07:44.370874 22076 raft_consensus.cc:2804] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:07:44.370946 22076 raft_consensus.cc:493] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:07:44.370963 22076 raft_consensus.cc:3060] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:44.371621 22076 raft_consensus.cc:515] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:44.371775 22076 leader_election.cc:290] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604 [CANDIDATE]: Term 1 election: Requested vote from peers 4765facb107747888bcb47b17955a2b8 (127.19.84.1:46445), 0e114f2373e14771981120579a59e39d (127.19.84.2:40287)
I20260131 03:07:44.372004 21859 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71d25857034c4f9c8c90119d5a3ac48b" candidate_uuid: "7a3f75adeeeb4c008b60ba490d012604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e114f2373e14771981120579a59e39d"
I20260131 03:07:44.372043 21716 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71d25857034c4f9c8c90119d5a3ac48b" candidate_uuid: "7a3f75adeeeb4c008b60ba490d012604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4765facb107747888bcb47b17955a2b8"
I20260131 03:07:44.372104 21859 raft_consensus.cc:3060] T 71d25857034c4f9c8c90119d5a3ac48b P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:44.372145 21716 raft_consensus.cc:3060] T 71d25857034c4f9c8c90119d5a3ac48b P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:44.372937 21716 raft_consensus.cc:2468] T 71d25857034c4f9c8c90119d5a3ac48b P 4765facb107747888bcb47b17955a2b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a3f75adeeeb4c008b60ba490d012604 in term 1.
I20260131 03:07:44.372947 21859 raft_consensus.cc:2468] T 71d25857034c4f9c8c90119d5a3ac48b P 0e114f2373e14771981120579a59e39d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a3f75adeeeb4c008b60ba490d012604 in term 1.
I20260131 03:07:44.373138 21935 leader_election.cc:304] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4765facb107747888bcb47b17955a2b8, 7a3f75adeeeb4c008b60ba490d012604; no voters:
I20260131 03:07:44.373261 22076 raft_consensus.cc:2804] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:07:44.373351 22076 raft_consensus.cc:697] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604 [term 1 LEADER]: Becoming Leader. State: Replica: 7a3f75adeeeb4c008b60ba490d012604, State: Running, Role: LEADER
I20260131 03:07:44.373415 22076 consensus_queue.cc:237] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:44.374169 21556 catalog_manager.cc:5654] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a3f75adeeeb4c008b60ba490d012604 (127.19.84.3). New cstate: current_term: 1 leader_uuid: "7a3f75adeeeb4c008b60ba490d012604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } health_report { overall_health: HEALTHY } } }
I20260131 03:07:44.464663 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:44.462472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46474 (local address 127.19.84.3:38153)
0131 03:07:44.462625 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:44.462626 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:44.462638 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:07:44.462702 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:44.462703 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:44.462712 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:44.462768 (+ 56us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:07:44.463011 (+ 243us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:44.463532 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:44.464171 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:44.464291 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:44.464308 (+ 17us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:07:44.464330 (+ 22us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:07:44.464378 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:44.464466 (+ 88us) server_negotiation.cc:300] Negotiation successful
0131 03:07:44.464488 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260131 03:07:44.467427 21716 raft_consensus.cc:1275] T 71d25857034c4f9c8c90119d5a3ac48b P 4765facb107747888bcb47b17955a2b8 [term 1 FOLLOWER]: Refusing update from remote peer 7a3f75adeeeb4c008b60ba490d012604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:07:44.467418 21859 raft_consensus.cc:1275] T 71d25857034c4f9c8c90119d5a3ac48b P 0e114f2373e14771981120579a59e39d [term 1 FOLLOWER]: Refusing update from remote peer 7a3f75adeeeb4c008b60ba490d012604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:07:44.467737 22076 consensus_queue.cc:1048] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:07:44.467880 22140 consensus_queue.cc:1048] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:07:44.469013 22128 mvcc.cc:204] Tried to move back new op lower bound from 7249219028856950784 to 7249219028473761792. Current Snapshot: MvccSnapshot[applied={T|T < 7249219028856950784}]
I20260131 03:07:44.477658 21556 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:07:44.477748 21556 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:07:44.478374 21556 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537: Sending DeleteTablet for 3 replicas of tablet 71d25857034c4f9c8c90119d5a3ac48b
I20260131 03:07:44.478606 21695 tablet_service.cc:1552] Processing DeleteTablet for tablet 71d25857034c4f9c8c90119d5a3ac48b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:07:44 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:43038
I20260131 03:07:44.478659 21839 tablet_service.cc:1552] Processing DeleteTablet for tablet 71d25857034c4f9c8c90119d5a3ac48b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:07:44 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:38498
I20260131 03:07:44.478659 21981 tablet_service.cc:1552] Processing DeleteTablet for tablet 71d25857034c4f9c8c90119d5a3ac48b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:07:44 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:46458
I20260131 03:07:44.478722 22148 tablet_replica.cc:333] T 71d25857034c4f9c8c90119d5a3ac48b P 4765facb107747888bcb47b17955a2b8: stopping tablet replica
I20260131 03:07:44.478737 22149 tablet_replica.cc:333] T 71d25857034c4f9c8c90119d5a3ac48b P 0e114f2373e14771981120579a59e39d: stopping tablet replica
I20260131 03:07:44.478768 22148 raft_consensus.cc:2243] T 71d25857034c4f9c8c90119d5a3ac48b P 4765facb107747888bcb47b17955a2b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:07:44.478775 22149 raft_consensus.cc:2243] T 71d25857034c4f9c8c90119d5a3ac48b P 0e114f2373e14771981120579a59e39d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:07:44.478806 22149 raft_consensus.cc:2272] T 71d25857034c4f9c8c90119d5a3ac48b P 0e114f2373e14771981120579a59e39d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:07:44.478806 22148 raft_consensus.cc:2272] T 71d25857034c4f9c8c90119d5a3ac48b P 4765facb107747888bcb47b17955a2b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:07:44.478881 22150 tablet_replica.cc:333] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604: stopping tablet replica
I20260131 03:07:44.478935 22150 raft_consensus.cc:2243] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:07:44.479019 22150 raft_consensus.cc:2272] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:07:44.479249 22148 ts_tablet_manager.cc:1916] T 71d25857034c4f9c8c90119d5a3ac48b P 4765facb107747888bcb47b17955a2b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:07:44.479408 22149 ts_tablet_manager.cc:1916] T 71d25857034c4f9c8c90119d5a3ac48b P 0e114f2373e14771981120579a59e39d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:07:44.480489 21556 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:07:44.480697 21556 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:07:44.480890 22150 ts_tablet_manager.cc:1916] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:07:44.481033 22149 ts_tablet_manager.cc:1929] T 71d25857034c4f9c8c90119d5a3ac48b P 0e114f2373e14771981120579a59e39d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:07:44.481101 22149 log.cc:1199] T 71d25857034c4f9c8c90119d5a3ac48b P 0e114f2373e14771981120579a59e39d: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-1/wal/wals/71d25857034c4f9c8c90119d5a3ac48b
I20260131 03:07:44.481357 22149 ts_tablet_manager.cc:1950] T 71d25857034c4f9c8c90119d5a3ac48b P 0e114f2373e14771981120579a59e39d: Deleting consensus metadata
I20260131 03:07:44.481766 21543 catalog_manager.cc:4985] TS 0e114f2373e14771981120579a59e39d (127.19.84.2:40287): tablet 71d25857034c4f9c8c90119d5a3ac48b (table test-table [id=07143169f360403ba5e600bdba2c10ba]) successfully deleted
I20260131 03:07:44.482144 21695 tablet_service.cc:1505] Processing CreateTablet for tablet 0ff6a8500768412f91cce0e55cd6d39b (DEFAULT_TABLE table=test-table [id=9554316f862c43508ab05d6759772e55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:07:44.482138 21839 tablet_service.cc:1505] Processing CreateTablet for tablet 0ff6a8500768412f91cce0e55cd6d39b (DEFAULT_TABLE table=test-table [id=9554316f862c43508ab05d6759772e55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:07:44.482177 21981 tablet_service.cc:1505] Processing CreateTablet for tablet 0ff6a8500768412f91cce0e55cd6d39b (DEFAULT_TABLE table=test-table [id=9554316f862c43508ab05d6759772e55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:07:44.482242 21839 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ff6a8500768412f91cce0e55cd6d39b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:44.482275 21981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ff6a8500768412f91cce0e55cd6d39b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:44.482544 22150 ts_tablet_manager.cc:1929] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:07:44.482589 22150 log.cc:1199] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/71d25857034c4f9c8c90119d5a3ac48b
I20260131 03:07:44.482806 22150 ts_tablet_manager.cc:1950] T 71d25857034c4f9c8c90119d5a3ac48b P 7a3f75adeeeb4c008b60ba490d012604: Deleting consensus metadata
I20260131 03:07:44.483001 22148 ts_tablet_manager.cc:1929] T 71d25857034c4f9c8c90119d5a3ac48b P 4765facb107747888bcb47b17955a2b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:07:44.483027 21695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ff6a8500768412f91cce0e55cd6d39b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:44.483054 22148 log.cc:1199] T 71d25857034c4f9c8c90119d5a3ac48b P 4765facb107747888bcb47b17955a2b8: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-0/wal/wals/71d25857034c4f9c8c90119d5a3ac48b
I20260131 03:07:44.483244 22159 tablet_bootstrap.cc:492] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d: Bootstrap starting.
I20260131 03:07:44.483263 22148 ts_tablet_manager.cc:1950] T 71d25857034c4f9c8c90119d5a3ac48b P 4765facb107747888bcb47b17955a2b8: Deleting consensus metadata
I20260131 03:07:44.483639 21544 catalog_manager.cc:4985] TS 4765facb107747888bcb47b17955a2b8 (127.19.84.1:46445): tablet 71d25857034c4f9c8c90119d5a3ac48b (table test-table [id=07143169f360403ba5e600bdba2c10ba]) successfully deleted
I20260131 03:07:44.483687 22160 tablet_bootstrap.cc:492] T 0ff6a8500768412f91cce0e55cd6d39b P 7a3f75adeeeb4c008b60ba490d012604: Bootstrap starting.
I20260131 03:07:44.483937 22159 tablet_bootstrap.cc:654] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:44.484004 22160 tablet_bootstrap.cc:654] T 0ff6a8500768412f91cce0e55cd6d39b P 7a3f75adeeeb4c008b60ba490d012604: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:44.484387 21541 catalog_manager.cc:4985] TS 7a3f75adeeeb4c008b60ba490d012604 (127.19.84.3:38153): tablet 71d25857034c4f9c8c90119d5a3ac48b (table test-table [id=07143169f360403ba5e600bdba2c10ba]) successfully deleted
I20260131 03:07:44.484606 22161 tablet_bootstrap.cc:492] T 0ff6a8500768412f91cce0e55cd6d39b P 4765facb107747888bcb47b17955a2b8: Bootstrap starting.
I20260131 03:07:44.484767 22160 tablet_bootstrap.cc:492] T 0ff6a8500768412f91cce0e55cd6d39b P 7a3f75adeeeb4c008b60ba490d012604: No bootstrap required, opened a new log
I20260131 03:07:44.484802 22160 ts_tablet_manager.cc:1403] T 0ff6a8500768412f91cce0e55cd6d39b P 7a3f75adeeeb4c008b60ba490d012604: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:44.484822 22159 tablet_bootstrap.cc:492] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d: No bootstrap required, opened a new log
I20260131 03:07:44.484865 22159 ts_tablet_manager.cc:1403] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260131 03:07:44.484915 22160 raft_consensus.cc:359] T 0ff6a8500768412f91cce0e55cd6d39b P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:44.484962 22160 raft_consensus.cc:385] T 0ff6a8500768412f91cce0e55cd6d39b P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:44.484958 22161 tablet_bootstrap.cc:654] T 0ff6a8500768412f91cce0e55cd6d39b P 4765facb107747888bcb47b17955a2b8: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:44.484978 22160 raft_consensus.cc:740] T 0ff6a8500768412f91cce0e55cd6d39b P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a3f75adeeeb4c008b60ba490d012604, State: Initialized, Role: FOLLOWER
I20260131 03:07:44.485041 22160 consensus_queue.cc:260] T 0ff6a8500768412f91cce0e55cd6d39b P 7a3f75adeeeb4c008b60ba490d012604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:44.485028 22159 raft_consensus.cc:359] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:44.485095 22159 raft_consensus.cc:385] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:44.485122 22159 raft_consensus.cc:740] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e114f2373e14771981120579a59e39d, State: Initialized, Role: FOLLOWER
I20260131 03:07:44.485158 22160 ts_tablet_manager.cc:1434] T 0ff6a8500768412f91cce0e55cd6d39b P 7a3f75adeeeb4c008b60ba490d012604: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260131 03:07:44.485170 22159 consensus_queue.cc:260] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:44.485291 22159 ts_tablet_manager.cc:1434] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260131 03:07:44.485615 22161 tablet_bootstrap.cc:492] T 0ff6a8500768412f91cce0e55cd6d39b P 4765facb107747888bcb47b17955a2b8: No bootstrap required, opened a new log
I20260131 03:07:44.485677 22161 ts_tablet_manager.cc:1403] T 0ff6a8500768412f91cce0e55cd6d39b P 4765facb107747888bcb47b17955a2b8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:44.485814 22161 raft_consensus.cc:359] T 0ff6a8500768412f91cce0e55cd6d39b P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:44.485893 22161 raft_consensus.cc:385] T 0ff6a8500768412f91cce0e55cd6d39b P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:44.485914 22161 raft_consensus.cc:740] T 0ff6a8500768412f91cce0e55cd6d39b P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4765facb107747888bcb47b17955a2b8, State: Initialized, Role: FOLLOWER
I20260131 03:07:44.485952 22161 consensus_queue.cc:260] T 0ff6a8500768412f91cce0e55cd6d39b P 4765facb107747888bcb47b17955a2b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:44.486059 22161 ts_tablet_manager.cc:1434] T 0ff6a8500768412f91cce0e55cd6d39b P 4765facb107747888bcb47b17955a2b8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260131 03:07:44.491613 22165 raft_consensus.cc:493] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:07:44.491703 22165 raft_consensus.cc:515] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:44.492007 22165 leader_election.cc:290] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4765facb107747888bcb47b17955a2b8 (127.19.84.1:46445), 7a3f75adeeeb4c008b60ba490d012604 (127.19.84.3:38153)
I20260131 03:07:44.494889 22176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:44.492167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:46445 (local address 127.19.84.2:46529)
0131 03:07:44.492418 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:44.492430 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:44.492509 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:44.492710 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:44.492711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:44.492715 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:07:44.492990 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:44.492993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:44.494004 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:44.494005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:44.494687 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:44.494689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:44.494754 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:44.494758 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:07:44.494785 (+ 27us) client_negotiation.cc:240] Negotiation successful
0131 03:07:44.494813 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":67,"threads_started":1}
I20260131 03:07:44.495329 22177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:44.492273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:46529 (local address 127.19.84.1:46445)
0131 03:07:44.492420 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:44.492422 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:44.492512 (+ 90us) server_negotiation.cc:408] Connection header received
0131 03:07:44.492568 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:44.492569 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:44.492582 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:44.492657 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:07:44.493118 (+ 461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:44.493912 (+ 794us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:44.494785 (+ 873us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:44.495224 (+ 439us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:44.495234 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:44.495253 (+ 19us) server_negotiation.cc:300] Negotiation successful
0131 03:07:44.495278 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260131 03:07:44.495612 21716 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ff6a8500768412f91cce0e55cd6d39b" candidate_uuid: "0e114f2373e14771981120579a59e39d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4765facb107747888bcb47b17955a2b8" is_pre_election: true
I20260131 03:07:44.495718 21716 raft_consensus.cc:2468] T 0ff6a8500768412f91cce0e55cd6d39b P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e114f2373e14771981120579a59e39d in term 0.
I20260131 03:07:44.495937 21792 leader_election.cc:304] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e114f2373e14771981120579a59e39d, 4765facb107747888bcb47b17955a2b8; no voters:
I20260131 03:07:44.496088 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:44.492246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:38153 (local address 127.19.84.2:53241)
0131 03:07:44.492433 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:44.492443 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:44.492509 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:44.492710 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:44.492711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:44.492714 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:07:44.492990 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:44.492993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:44.495032 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:44.495033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:44.495872 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:44.495875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:44.495946 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:44.495950 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:07:44.495982 (+ 32us) client_negotiation.cc:240] Negotiation successful
0131 03:07:44.496016 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":142,"threads_started":1}
I20260131 03:07:44.496089 22165 raft_consensus.cc:2804] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:07:44.496234 22165 raft_consensus.cc:493] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:07:44.496259 22165 raft_consensus.cc:3060] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:44.496591 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:44.492272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:53241 (local address 127.19.84.3:38153)
0131 03:07:44.492347 (+ 75us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:44.492350 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:44.492513 (+ 163us) server_negotiation.cc:408] Connection header received
0131 03:07:44.492568 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:44.492569 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:44.492583 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:44.492657 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:07:44.493119 (+ 462us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:44.493912 (+ 793us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:44.495985 (+ 2073us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:44.496449 (+ 464us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:44.496458 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:44.496483 (+ 25us) server_negotiation.cc:300] Negotiation successful
0131 03:07:44.496531 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260131 03:07:44.496863 22001 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ff6a8500768412f91cce0e55cd6d39b" candidate_uuid: "0e114f2373e14771981120579a59e39d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a3f75adeeeb4c008b60ba490d012604" is_pre_election: true
I20260131 03:07:44.496934 22001 raft_consensus.cc:2468] T 0ff6a8500768412f91cce0e55cd6d39b P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e114f2373e14771981120579a59e39d in term 0.
I20260131 03:07:44.497040 22165 raft_consensus.cc:515] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:44.497224 22165 leader_election.cc:290] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d [CANDIDATE]: Term 1 election: Requested vote from peers 4765facb107747888bcb47b17955a2b8 (127.19.84.1:46445), 7a3f75adeeeb4c008b60ba490d012604 (127.19.84.3:38153)
I20260131 03:07:44.497565 22001 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ff6a8500768412f91cce0e55cd6d39b" candidate_uuid: "0e114f2373e14771981120579a59e39d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a3f75adeeeb4c008b60ba490d012604"
I20260131 03:07:44.497663 22001 raft_consensus.cc:3060] T 0ff6a8500768412f91cce0e55cd6d39b P 7a3f75adeeeb4c008b60ba490d012604 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:44.498258 22001 raft_consensus.cc:2468] T 0ff6a8500768412f91cce0e55cd6d39b P 7a3f75adeeeb4c008b60ba490d012604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e114f2373e14771981120579a59e39d in term 1.
I20260131 03:07:44.498395 21716 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ff6a8500768412f91cce0e55cd6d39b" candidate_uuid: "0e114f2373e14771981120579a59e39d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4765facb107747888bcb47b17955a2b8"
I20260131 03:07:44.498492 21716 raft_consensus.cc:3060] T 0ff6a8500768412f91cce0e55cd6d39b P 4765facb107747888bcb47b17955a2b8 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:44.498481 21789 leader_election.cc:304] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e114f2373e14771981120579a59e39d, 7a3f75adeeeb4c008b60ba490d012604; no voters:
I20260131 03:07:44.498565 22165 raft_consensus.cc:2804] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:07:44.498723 22165 raft_consensus.cc:697] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d [term 1 LEADER]: Becoming Leader. State: Replica: 0e114f2373e14771981120579a59e39d, State: Running, Role: LEADER
I20260131 03:07:44.498852 22165 consensus_queue.cc:237] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } }
I20260131 03:07:44.499415 21716 raft_consensus.cc:2468] T 0ff6a8500768412f91cce0e55cd6d39b P 4765facb107747888bcb47b17955a2b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e114f2373e14771981120579a59e39d in term 1.
I20260131 03:07:44.499590 21557 catalog_manager.cc:5654] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e114f2373e14771981120579a59e39d (127.19.84.2). New cstate: current_term: 1 leader_uuid: "0e114f2373e14771981120579a59e39d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e114f2373e14771981120579a59e39d" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 } health_report { overall_health: UNKNOWN } } }
I20260131 03:07:44.507436 22181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:44.505412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38508 (local address 127.19.84.2:40287)
0131 03:07:44.505594 (+ 182us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:44.505596 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:44.505607 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:07:44.505639 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:44.505640 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:44.505661 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:44.505721 (+ 60us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:07:44.505935 (+ 214us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:44.506433 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:44.506944 (+ 511us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:44.507067 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:44.507097 (+ 30us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:07:44.507188 (+ 91us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:07:44.507250 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:44.507301 (+ 51us) server_negotiation.cc:300] Negotiation successful
0131 03:07:44.507319 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":106,"threads_started":1}
I20260131 03:07:44.511575 22177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:44.509676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:58387 (local address 127.19.84.1:46445)
0131 03:07:44.509734 (+ 58us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:44.509736 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:44.509744 (+ 8us) server_negotiation.cc:408] Connection header received
0131 03:07:44.509781 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:44.509781 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:44.509789 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:44.509827 (+ 38us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:07:44.510071 (+ 244us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:44.510507 (+ 436us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:44.511107 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:44.511247 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:44.511264 (+ 17us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:07:44.511285 (+ 21us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:07:44.511331 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:44.511493 (+ 162us) server_negotiation.cc:300] Negotiation successful
0131 03:07:44.511522 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260131 03:07:44.511587 22184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:44.509509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:46445 (local address 127.19.84.2:58387)
0131 03:07:44.509676 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:44.509689 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:44.509736 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:44.509844 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:44.509844 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:44.509846 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:07:44.510013 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:44.510014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:44.510601 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:44.510602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:44.511024 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:44.511026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:44.511090 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:44.511105 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:07:44.511355 (+ 250us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:07:44.511356 (+ 1us) client_negotiation.cc:769] Sending connection context
0131 03:07:44.511474 (+ 118us) client_negotiation.cc:240] Negotiation successful
0131 03:07:44.511519 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260131 03:07:44.514024 22001 raft_consensus.cc:1275] T 0ff6a8500768412f91cce0e55cd6d39b P 7a3f75adeeeb4c008b60ba490d012604 [term 1 FOLLOWER]: Refusing update from remote peer 0e114f2373e14771981120579a59e39d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:07:44.514065 21716 raft_consensus.cc:1275] T 0ff6a8500768412f91cce0e55cd6d39b P 4765facb107747888bcb47b17955a2b8 [term 1 FOLLOWER]: Refusing update from remote peer 0e114f2373e14771981120579a59e39d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:07:44.514271 22179 consensus_queue.cc:1048] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a3f75adeeeb4c008b60ba490d012604" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 38153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:07:44.514437 22179 consensus_queue.cc:1048] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4765facb107747888bcb47b17955a2b8" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:07:44.521083 22181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:44.518674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:41945 (local address 127.19.84.2:40287)
0131 03:07:44.518718 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:44.518720 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:44.518849 (+ 129us) server_negotiation.cc:408] Connection header received
0131 03:07:44.518921 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:44.518922 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:44.518934 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:44.518988 (+ 54us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:07:44.519251 (+ 263us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:44.520052 (+ 801us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:44.520651 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:44.520801 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:44.520823 (+ 22us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:07:44.520843 (+ 20us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:07:44.520905 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:44.520993 (+ 88us) server_negotiation.cc:300] Negotiation successful
0131 03:07:44.521016 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260131 03:07:44.521107 22188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:44.518620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:40287 (local address 127.19.84.1:41945)
0131 03:07:44.518768 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:44.518778 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:44.518863 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:44.518991 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:44.518992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:44.518994 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:07:44.519172 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:44.519173 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:44.520142 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:44.520143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:44.520550 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:44.520552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:44.520643 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:44.520648 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:07:44.520921 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:07:44.520923 (+ 2us) client_negotiation.cc:769] Sending connection context
0131 03:07:44.520990 (+ 67us) client_negotiation.cc:240] Negotiation successful
0131 03:07:44.521038 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
W20260131 03:07:44.521469 22132 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20260131 03:07:46.045410 21556 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:07:46.045508 21556 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:07:46.046317 21556 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 11a07d3643e3429591c371e578821537: Sending DeleteTablet for 3 replicas of tablet 0ff6a8500768412f91cce0e55cd6d39b
I20260131 03:07:46.046587 21981 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ff6a8500768412f91cce0e55cd6d39b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:07:46 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:46458
I20260131 03:07:46.046605 21695 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ff6a8500768412f91cce0e55cd6d39b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:07:46 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:43038
I20260131 03:07:46.046654 21839 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ff6a8500768412f91cce0e55cd6d39b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:07:46 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:38498
I20260131 03:07:46.046833 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 21631
I20260131 03:07:46.046870 22213 tablet_replica.cc:333] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d: stopping tablet replica
I20260131 03:07:46.046936 22213 raft_consensus.cc:2243] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:07:46.047574 22213 raft_consensus.cc:2272] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:07:46.047901 22213 ts_tablet_manager.cc:1916] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:07:46.047931 22211 tablet_replica.cc:333] T 0ff6a8500768412f91cce0e55cd6d39b P 7a3f75adeeeb4c008b60ba490d012604: stopping tablet replica
I20260131 03:07:46.048058 22211 raft_consensus.cc:2243] T 0ff6a8500768412f91cce0e55cd6d39b P 7a3f75adeeeb4c008b60ba490d012604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:07:46.048159 22211 raft_consensus.cc:2272] T 0ff6a8500768412f91cce0e55cd6d39b P 7a3f75adeeeb4c008b60ba490d012604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:07:46.048444 22211 ts_tablet_manager.cc:1916] T 0ff6a8500768412f91cce0e55cd6d39b P 7a3f75adeeeb4c008b60ba490d012604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:07:46.049918 22213 ts_tablet_manager.cc:1929] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260131 03:07:46.050005 22213 log.cc:1199] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-1/wal/wals/0ff6a8500768412f91cce0e55cd6d39b
I20260131 03:07:46.050352 22213 ts_tablet_manager.cc:1950] T 0ff6a8500768412f91cce0e55cd6d39b P 0e114f2373e14771981120579a59e39d: Deleting consensus metadata
I20260131 03:07:46.050586 22211 ts_tablet_manager.cc:1929] T 0ff6a8500768412f91cce0e55cd6d39b P 7a3f75adeeeb4c008b60ba490d012604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260131 03:07:46.050684 22211 log.cc:1199] T 0ff6a8500768412f91cce0e55cd6d39b P 7a3f75adeeeb4c008b60ba490d012604: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/0ff6a8500768412f91cce0e55cd6d39b
I20260131 03:07:46.050753 21543 catalog_manager.cc:4985] TS 0e114f2373e14771981120579a59e39d (127.19.84.2:40287): tablet 0ff6a8500768412f91cce0e55cd6d39b (table test-table [id=9554316f862c43508ab05d6759772e55]) successfully deleted
I20260131 03:07:46.050978 22211 ts_tablet_manager.cc:1950] T 0ff6a8500768412f91cce0e55cd6d39b P 7a3f75adeeeb4c008b60ba490d012604: Deleting consensus metadata
I20260131 03:07:46.051340 21541 catalog_manager.cc:4985] TS 7a3f75adeeeb4c008b60ba490d012604 (127.19.84.3:38153): tablet 0ff6a8500768412f91cce0e55cd6d39b (table test-table [id=9554316f862c43508ab05d6759772e55]) successfully deleted
W20260131 03:07:46.054822 21544 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.84.1:46445 (error 108)
I20260131 03:07:46.055121 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 21773
I20260131 03:07:46.055913 22215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:46.055210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:46445 (local address 127.0.0.1:43074)
0131 03:07:46.055809 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:46.055853 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.1:46445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":59,"threads_started":1}
W20260131 03:07:46.055997 21544 catalog_manager.cc:4712] TS 4765facb107747888bcb47b17955a2b8 (127.19.84.1:46445): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0ff6a8500768412f91cce0e55cd6d39b: Network error: Client connection negotiation failed: client connection to 127.19.84.1:46445: connect: Connection refused (error 111)
I20260131 03:07:46.062347 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 21916
I20260131 03:07:46.067975 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 21525
2026-01-31T03:07:46Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7006 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:07:46 dist-test-slave-5nnp krb5kdc[22218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:07:46 dist-test-slave-5nnp krb5kdc[22218](info): set up 2 sockets
Jan 31 03:07:46 dist-test-slave-5nnp krb5kdc[22218](info): commencing operation
krb5kdc: starting...
W20260131 03:07:48.088464 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.999s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:07:48 dist-test-slave-5nnp krb5kdc[22218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:07:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:07:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:48.221992 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:45883
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:45883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:48.300575 22234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:48.300809 22234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:48.300843 22234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:48.302266 22234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:07:48.302353 22234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:48.302376 22234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:48.302397 22234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:07:48.302423 22234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:07:48.304076 22234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:45883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:45883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:48.304340 22234 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:48.304538 22234 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:48.307575 22239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:48.307592 22240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:48.307817 22242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:48.308089 22234 server_base.cc:1060] running on GCE node
I20260131 03:07:48.308408 22234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:48.308816 22234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:48.310001 22234 hybrid_clock.cc:648] HybridClock initialized: now 1769828868309986 us; error 35 us; skew 500 ppm
Jan 31 03:07:48 dist-test-slave-5nnp krb5kdc[22218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828868, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:48.312659 22234 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:07:48.313446 22234 webserver.cc:492] Webserver started at http://127.19.84.62:45247/ using document root <none> and password file <none>
I20260131 03:07:48.313684 22234 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:48.313746 22234 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:48.313879 22234 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:48.314847 22234 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "290bf14dad154ad4b18f8e4d522ac944"
format_stamp: "Formatted at 2026-01-31 03:07:48 on dist-test-slave-5nnp"
server_key: "fd8348c19ac39a18bed6b8d91287f5ee"
server_key_iv: "b97750327728bd3608e3b6a0129c2a5f"
server_key_version: "encryptionkey@0"
I20260131 03:07:48.315173 22234 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "290bf14dad154ad4b18f8e4d522ac944"
format_stamp: "Formatted at 2026-01-31 03:07:48 on dist-test-slave-5nnp"
server_key: "fd8348c19ac39a18bed6b8d91287f5ee"
server_key_iv: "b97750327728bd3608e3b6a0129c2a5f"
server_key_version: "encryptionkey@0"
I20260131 03:07:48.316449 22234 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:07:48.317245 22249 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:48.317464 22234 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:48.317566 22234 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "290bf14dad154ad4b18f8e4d522ac944"
format_stamp: "Formatted at 2026-01-31 03:07:48 on dist-test-slave-5nnp"
server_key: "fd8348c19ac39a18bed6b8d91287f5ee"
server_key_iv: "b97750327728bd3608e3b6a0129c2a5f"
server_key_version: "encryptionkey@0"
I20260131 03:07:48.317646 22234 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:48.324461 22234 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:48.326555 22234 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:48.326719 22234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:48.331468 22234 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:45883
I20260131 03:07:48.331542 22301 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:45883 every 8 connection(s)
I20260131 03:07:48.331893 22234 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:07:48.332496 22302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:48.335029 22302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 290bf14dad154ad4b18f8e4d522ac944: Bootstrap starting.
I20260131 03:07:48.335672 22302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 290bf14dad154ad4b18f8e4d522ac944: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:48.335944 22302 log.cc:826] T 00000000000000000000000000000000 P 290bf14dad154ad4b18f8e4d522ac944: Log is configured to *not* fsync() on all Append() calls
I20260131 03:07:48.336452 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 22234
I20260131 03:07:48.336584 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:07:48.336752 19792 external_mini_cluster.cc:1468] Setting key d7a962ebb0e9b03294fc92f338addfc4
I20260131 03:07:48.336874 22302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 290bf14dad154ad4b18f8e4d522ac944: No bootstrap required, opened a new log
I20260131 03:07:48.338305 22302 raft_consensus.cc:359] T 00000000000000000000000000000000 P 290bf14dad154ad4b18f8e4d522ac944 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290bf14dad154ad4b18f8e4d522ac944" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 45883 } }
I20260131 03:07:48.338451 22302 raft_consensus.cc:385] T 00000000000000000000000000000000 P 290bf14dad154ad4b18f8e4d522ac944 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:48.338507 22302 raft_consensus.cc:740] T 00000000000000000000000000000000 P 290bf14dad154ad4b18f8e4d522ac944 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290bf14dad154ad4b18f8e4d522ac944, State: Initialized, Role: FOLLOWER
I20260131 03:07:48.338646 22302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 290bf14dad154ad4b18f8e4d522ac944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290bf14dad154ad4b18f8e4d522ac944" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 45883 } }
I20260131 03:07:48.338719 22302 raft_consensus.cc:399] T 00000000000000000000000000000000 P 290bf14dad154ad4b18f8e4d522ac944 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:07:48.338742 22302 raft_consensus.cc:493] T 00000000000000000000000000000000 P 290bf14dad154ad4b18f8e4d522ac944 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:07:48.338789 22302 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 290bf14dad154ad4b18f8e4d522ac944 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:48.339344 22302 raft_consensus.cc:515] T 00000000000000000000000000000000 P 290bf14dad154ad4b18f8e4d522ac944 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290bf14dad154ad4b18f8e4d522ac944" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 45883 } }
I20260131 03:07:48.339501 22302 leader_election.cc:304] T 00000000000000000000000000000000 P 290bf14dad154ad4b18f8e4d522ac944 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 290bf14dad154ad4b18f8e4d522ac944; no voters:
I20260131 03:07:48.339677 22302 leader_election.cc:290] T 00000000000000000000000000000000 P 290bf14dad154ad4b18f8e4d522ac944 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:07:48.339727 22307 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 290bf14dad154ad4b18f8e4d522ac944 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:07:48.339919 22307 raft_consensus.cc:697] T 00000000000000000000000000000000 P 290bf14dad154ad4b18f8e4d522ac944 [term 1 LEADER]: Becoming Leader. State: Replica: 290bf14dad154ad4b18f8e4d522ac944, State: Running, Role: LEADER
I20260131 03:07:48.339943 22302 sys_catalog.cc:565] T 00000000000000000000000000000000 P 290bf14dad154ad4b18f8e4d522ac944 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:07:48.340075 22307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 290bf14dad154ad4b18f8e4d522ac944 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290bf14dad154ad4b18f8e4d522ac944" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 45883 } }
I20260131 03:07:48.341353 22309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 290bf14dad154ad4b18f8e4d522ac944 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "290bf14dad154ad4b18f8e4d522ac944" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290bf14dad154ad4b18f8e4d522ac944" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 45883 } } }
I20260131 03:07:48.341810 22309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 290bf14dad154ad4b18f8e4d522ac944 [sys.catalog]: This master's current role is: LEADER
I20260131 03:07:48.342840 22320 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:07:48.342824 22308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 290bf14dad154ad4b18f8e4d522ac944 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 290bf14dad154ad4b18f8e4d522ac944. Latest consensus state: current_term: 1 leader_uuid: "290bf14dad154ad4b18f8e4d522ac944" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290bf14dad154ad4b18f8e4d522ac944" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 45883 } } }
I20260131 03:07:48.342973 22308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 290bf14dad154ad4b18f8e4d522ac944 [sys.catalog]: This master's current role is: LEADER
Jan 31 03:07:48 dist-test-slave-5nnp krb5kdc[22218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:48.343544 22320 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:07:48.345804 22320 catalog_manager.cc:1357] Generated new cluster ID: 7aba1b6fdb664a4093c0268b3272ad24
I20260131 03:07:48.345875 22320 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:07:48.347738 22306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:48.337285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43892 (local address 127.19.84.62:45883)
0131 03:07:48.337582 (+ 297us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:48.337591 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:48.337612 (+ 21us) server_negotiation.cc:408] Connection header received
0131 03:07:48.338314 (+ 702us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:48.338322 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:48.338371 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:48.338523 (+ 152us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:48.339408 (+ 885us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:48.340538 (+ 1130us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:48.341162 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:48.341409 (+ 247us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:48.343892 (+ 2483us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:48.343902 (+ 10us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:48.343902 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:48.343927 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:48.346236 (+ 2309us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:48.346651 (+ 415us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:48.346654 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:48.346654 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:48.346715 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:48.346891 (+ 176us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:48.346892 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:48.346892 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:48.347135 (+ 243us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:48.347273 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:48.347443 (+ 170us) server_negotiation.cc:300] Negotiation successful
0131 03:07:48.347567 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":221,"threads_started":1}
I20260131 03:07:48.364429 22320 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:07:48.365213 22320 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:07:48.372895 22320 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 290bf14dad154ad4b18f8e4d522ac944: Generated new TSK 0
I20260131 03:07:48.373132 22320 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:48.434306 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:45883
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:48.525007 22330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:48.525214 22330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:48.525255 22330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:48.526796 22330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:48.526886 22330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:48.526932 22330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:07:48.528733 22330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:45883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:48.529008 22330 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:48.529215 22330 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:48.531965 22338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:48.531965 22335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:48.531989 22336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:48.531991 22330 server_base.cc:1060] running on GCE node
I20260131 03:07:48.532299 22330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:48.532474 22330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:48.533638 22330 hybrid_clock.cc:648] HybridClock initialized: now 1769828868533618 us; error 40 us; skew 500 ppm
Jan 31 03:07:48 dist-test-slave-5nnp krb5kdc[22218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828868, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:48.537443 22330 init.cc:377] Logged in from keytab as kudu/127.19.84.1@KRBTEST.COM (short username kudu)
I20260131 03:07:48.538126 22330 webserver.cc:492] Webserver started at http://127.19.84.1:45405/ using document root <none> and password file <none>
I20260131 03:07:48.538336 22330 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:48.538393 22330 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:48.538522 22330 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:48.539597 22330 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "4ba9b43240484cbfb9cf5e1adfbb11ae"
format_stamp: "Formatted at 2026-01-31 03:07:48 on dist-test-slave-5nnp"
server_key: "d567452f00f2be285c2da7d7cef6624f"
server_key_iv: "0e7da932a7c35d50fbea8a27be6877c5"
server_key_version: "encryptionkey@0"
I20260131 03:07:48.539932 22330 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "4ba9b43240484cbfb9cf5e1adfbb11ae"
format_stamp: "Formatted at 2026-01-31 03:07:48 on dist-test-slave-5nnp"
server_key: "d567452f00f2be285c2da7d7cef6624f"
server_key_iv: "0e7da932a7c35d50fbea8a27be6877c5"
server_key_version: "encryptionkey@0"
I20260131 03:07:48.541227 22330 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:48.541950 22345 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:48.542179 22330 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:48.542265 22330 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "4ba9b43240484cbfb9cf5e1adfbb11ae"
format_stamp: "Formatted at 2026-01-31 03:07:48 on dist-test-slave-5nnp"
server_key: "d567452f00f2be285c2da7d7cef6624f"
server_key_iv: "0e7da932a7c35d50fbea8a27be6877c5"
server_key_version: "encryptionkey@0"
I20260131 03:07:48.542342 22330 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:48.563552 22330 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:48.565658 22330 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:48.565812 22330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:48.566056 22330 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:07:48.566385 22330 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:07:48.566444 22330 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:48.566486 22330 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:07:48.566519 22330 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:48.572331 22330 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:36029
I20260131 03:07:48.572436 22458 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:36029 every 8 connection(s)
I20260131 03:07:48.572706 22330 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
Jan 31 03:07:48 dist-test-slave-5nnp krb5kdc[22218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828868, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:48.579862 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 22330
I20260131 03:07:48.579970 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:07:48.580144 19792 external_mini_cluster.cc:1468] Setting key ff4d6f052ad8940276078dfde4dc4865
I20260131 03:07:48.581429 22461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:48.573374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:45883 (local address 127.19.84.1:60177)
0131 03:07:48.573519 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:48.573561 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:48.574033 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:48.574319 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:48.574323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:48.574640 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:48.574923 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:48.574926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:48.575695 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:48.575696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:48.576099 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:48.576101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:48.576451 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:48.576939 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:48.576956 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:48.578297 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:48.580011 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:48.580017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:48.580020 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:48.580279 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:48.580483 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:48.580484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:48.580485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:48.580567 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:48.580972 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:48.580977 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:48.581092 (+ 115us) client_negotiation.cc:769] Sending connection context
0131 03:07:48.581185 (+ 93us) client_negotiation.cc:240] Negotiation successful
0131 03:07:48.581290 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260131 03:07:48.581512 22306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:48.573420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:60177 (local address 127.19.84.62:45883)
0131 03:07:48.573465 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:48.573467 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:48.573628 (+ 161us) server_negotiation.cc:408] Connection header received
0131 03:07:48.574108 (+ 480us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:48.574109 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:48.574123 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:48.574185 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:48.575015 (+ 830us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:48.575604 (+ 589us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:48.576177 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:48.576294 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:48.578380 (+ 2086us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:48.578383 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:48.578384 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:48.578390 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:48.579869 (+ 1479us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:48.580366 (+ 497us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:48.580367 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:48.580367 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:48.580412 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:48.580673 (+ 261us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:48.580674 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:48.580675 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:48.580891 (+ 216us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:48.580980 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:48.581237 (+ 257us) server_negotiation.cc:300] Negotiation successful
0131 03:07:48.581355 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260131 03:07:48.582755 22459 heartbeater.cc:344] Connected to a master server at 127.19.84.62:45883
I20260131 03:07:48.582885 22459 heartbeater.cc:461] Registering TS with master...
I20260131 03:07:48.583140 22459 heartbeater.cc:507] Master 127.19.84.62:45883 requested a full tablet report, sending...
I20260131 03:07:48.583674 22266 ts_manager.cc:194] Registered new tserver with Master: 4ba9b43240484cbfb9cf5e1adfbb11ae (127.19.84.1:36029)
I20260131 03:07:48.584322 22266 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.1@KRBTEST.COM'} at 127.19.84.1:60177
WARNING: no policy specified for kudu/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:48.627897 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:45883
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:48.707142 22466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:48.707365 22466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:48.707408 22466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:48.708866 22466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:48.708941 22466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:48.708982 22466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:07:48.710700 22466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:45883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:48.710968 22466 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:48.711160 22466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:48.713891 22472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:48.713963 22471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:48.713992 22466 server_base.cc:1060] running on GCE node
W20260131 03:07:48.714067 22474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:48.714329 22466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:48.714592 22466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:48.715793 22466 hybrid_clock.cc:648] HybridClock initialized: now 1769828868715774 us; error 66 us; skew 500 ppm
Jan 31 03:07:48 dist-test-slave-5nnp krb5kdc[22218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828868, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:48.718416 22466 init.cc:377] Logged in from keytab as kudu/127.19.84.2@KRBTEST.COM (short username kudu)
I20260131 03:07:48.718983 22466 webserver.cc:492] Webserver started at http://127.19.84.2:40953/ using document root <none> and password file <none>
I20260131 03:07:48.719190 22466 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:48.719247 22466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:48.719431 22466 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:48.720374 22466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "1f65fd42de5a48858f4976ddb28f250c"
format_stamp: "Formatted at 2026-01-31 03:07:48 on dist-test-slave-5nnp"
server_key: "646279f7f473b4c1fc693ace75985792"
server_key_iv: "b2c7f43a6c630f00baf74de86f71d8b5"
server_key_version: "encryptionkey@0"
I20260131 03:07:48.720768 22466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "1f65fd42de5a48858f4976ddb28f250c"
format_stamp: "Formatted at 2026-01-31 03:07:48 on dist-test-slave-5nnp"
server_key: "646279f7f473b4c1fc693ace75985792"
server_key_iv: "b2c7f43a6c630f00baf74de86f71d8b5"
server_key_version: "encryptionkey@0"
I20260131 03:07:48.722008 22466 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260131 03:07:48.722849 22481 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:48.723143 22466 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:07:48.723227 22466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "1f65fd42de5a48858f4976ddb28f250c"
format_stamp: "Formatted at 2026-01-31 03:07:48 on dist-test-slave-5nnp"
server_key: "646279f7f473b4c1fc693ace75985792"
server_key_iv: "b2c7f43a6c630f00baf74de86f71d8b5"
server_key_version: "encryptionkey@0"
I20260131 03:07:48.723313 22466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:48.733237 22466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:48.735496 22466 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:48.735809 22466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:48.736116 22466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:07:48.736483 22466 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:07:48.736544 22466 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:48.736593 22466 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:07:48.736622 22466 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:48.742895 22466 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:45639
I20260131 03:07:48.742975 22594 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:45639 every 8 connection(s)
I20260131 03:07:48.743289 22466 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
Jan 31 03:07:48 dist-test-slave-5nnp krb5kdc[22218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828868, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:48.752193 22306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:48.744211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:60985 (local address 127.19.84.62:45883)
0131 03:07:48.744300 (+ 89us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:48.744302 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:48.744447 (+ 145us) server_negotiation.cc:408] Connection header received
0131 03:07:48.744999 (+ 552us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:48.745001 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:48.745014 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:48.745070 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:48.745848 (+ 778us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:48.746472 (+ 624us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:48.747072 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:48.747265 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:48.749506 (+ 2241us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:48.749511 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:48.749512 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:48.749526 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:48.750798 (+ 1272us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:48.751215 (+ 417us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:48.751217 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:48.751217 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:48.751250 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:48.751522 (+ 272us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:48.751523 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:48.751524 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:48.751632 (+ 108us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:48.751707 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:48.751991 (+ 284us) server_negotiation.cc:300] Negotiation successful
0131 03:07:48.752094 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260131 03:07:48.752312 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:48.744108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:45883 (local address 127.19.84.2:60985)
0131 03:07:48.744327 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:48.744365 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:48.744887 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:48.745144 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:48.745147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:48.745469 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:48.745746 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:48.745748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:48.746580 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:48.746581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:48.746983 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:48.746984 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:48.747471 (+ 487us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:48.747965 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:48.747982 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:48.749406 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:48.750918 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:48.750924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:48.750927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:48.751137 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:48.751317 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:48.751318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:48.751318 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:48.751449 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:48.751708 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:48.751713 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:48.751846 (+ 133us) client_negotiation.cc:769] Sending connection context
0131 03:07:48.751972 (+ 126us) client_negotiation.cc:240] Negotiation successful
0131 03:07:48.752114 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":102,"threads_started":1}
I20260131 03:07:48.752206 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 22466
I20260131 03:07:48.752427 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:07:48.752658 19792 external_mini_cluster.cc:1468] Setting key 4e4853ddde599eebd64310e45fb27db8
I20260131 03:07:48.752884 22595 heartbeater.cc:344] Connected to a master server at 127.19.84.62:45883
I20260131 03:07:48.752990 22595 heartbeater.cc:461] Registering TS with master...
I20260131 03:07:48.753233 22595 heartbeater.cc:507] Master 127.19.84.62:45883 requested a full tablet report, sending...
I20260131 03:07:48.753710 22266 ts_manager.cc:194] Registered new tserver with Master: 1f65fd42de5a48858f4976ddb28f250c (127.19.84.2:45639)
I20260131 03:07:48.754145 22266 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.2@KRBTEST.COM'} at 127.19.84.2:60985
WARNING: no policy specified for kudu/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:48.803161 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:45883
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:48.883831 22602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:48.883991 22602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:48.884014 22602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:48.885353 22602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:48.885394 22602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:48.885421 22602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:07:48.886932 22602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:45883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:48.887159 22602 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:48.887365 22602 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:48.890327 22610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:48.890327 22608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:48.890563 22602 server_base.cc:1060] running on GCE node
W20260131 03:07:48.890316 22607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:48.891010 22602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:48.891263 22602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:48.892416 22602 hybrid_clock.cc:648] HybridClock initialized: now 1769828868892410 us; error 38 us; skew 500 ppm
Jan 31 03:07:48 dist-test-slave-5nnp krb5kdc[22218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828868, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:48.894872 22602 init.cc:377] Logged in from keytab as kudu/127.19.84.3@KRBTEST.COM (short username kudu)
I20260131 03:07:48.895489 22602 webserver.cc:492] Webserver started at http://127.19.84.3:43381/ using document root <none> and password file <none>
I20260131 03:07:48.895658 22602 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:48.895689 22602 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:48.895773 22602 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:48.896670 22602 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "2deb229679d44393a49c9c752d57affd"
format_stamp: "Formatted at 2026-01-31 03:07:48 on dist-test-slave-5nnp"
server_key: "08b0e292116042cdd5930b9a6b61f6ec"
server_key_iv: "6c53e6df59c30bddde4525f9543830ea"
server_key_version: "encryptionkey@0"
I20260131 03:07:48.896939 22602 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "2deb229679d44393a49c9c752d57affd"
format_stamp: "Formatted at 2026-01-31 03:07:48 on dist-test-slave-5nnp"
server_key: "08b0e292116042cdd5930b9a6b61f6ec"
server_key_iv: "6c53e6df59c30bddde4525f9543830ea"
server_key_version: "encryptionkey@0"
I20260131 03:07:48.898053 22602 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260131 03:07:48.898798 22617 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:48.899010 22602 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:07:48.899086 22602 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "2deb229679d44393a49c9c752d57affd"
format_stamp: "Formatted at 2026-01-31 03:07:48 on dist-test-slave-5nnp"
server_key: "08b0e292116042cdd5930b9a6b61f6ec"
server_key_iv: "6c53e6df59c30bddde4525f9543830ea"
server_key_version: "encryptionkey@0"
I20260131 03:07:48.899138 22602 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:48.931214 22602 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:48.933233 22602 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:48.933346 22602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:48.933581 22602 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:07:48.933868 22602 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:07:48.933897 22602 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:48.933920 22602 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:07:48.933933 22602 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:48.939561 22602 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:37585
I20260131 03:07:48.939682 22730 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:37585 every 8 connection(s)
I20260131 03:07:48.939949 22602 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
Jan 31 03:07:48 dist-test-slave-5nnp krb5kdc[22218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828868, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:48.947980 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 22602
I20260131 03:07:48.948050 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:07:48.948218 19792 external_mini_cluster.cc:1468] Setting key 229ac8b83b4a68e7ffb921b0414bdcc6
I20260131 03:07:48.949498 22306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:48.940790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:56687 (local address 127.19.84.62:45883)
0131 03:07:48.940836 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:48.940838 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:48.941027 (+ 189us) server_negotiation.cc:408] Connection header received
0131 03:07:48.941553 (+ 526us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:48.941554 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:48.941567 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:48.941647 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:48.942414 (+ 767us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:48.943023 (+ 609us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:48.943702 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:48.943880 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:48.946239 (+ 2359us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:48.946244 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:48.946244 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:48.946254 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:48.947868 (+ 1614us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:48.948361 (+ 493us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:48.948362 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:48.948363 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:48.948393 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:48.948659 (+ 266us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:48.948660 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:48.948660 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:48.948797 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:48.948867 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:48.949277 (+ 410us) server_negotiation.cc:300] Negotiation successful
0131 03:07:48.949402 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260131 03:07:48.949548 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:48.940764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:45883 (local address 127.19.84.3:56687)
0131 03:07:48.940904 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:48.940942 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:48.941446 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:48.941711 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:48.941715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:48.942037 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:48.942313 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:48.942316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:48.943156 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:48.943157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:48.943607 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:48.943608 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:48.944070 (+ 462us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:48.944599 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:48.944619 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:48.946125 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:48.947986 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:48.947993 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:48.947996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:48.948288 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:48.948522 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:48.948523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:48.948523 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:48.948592 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:48.948875 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:48.948882 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:48.949064 (+ 182us) client_negotiation.cc:769] Sending connection context
0131 03:07:48.949198 (+ 134us) client_negotiation.cc:240] Negotiation successful
0131 03:07:48.949382 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260131 03:07:48.950045 22731 heartbeater.cc:344] Connected to a master server at 127.19.84.62:45883
I20260131 03:07:48.950155 22731 heartbeater.cc:461] Registering TS with master...
I20260131 03:07:48.950369 22731 heartbeater.cc:507] Master 127.19.84.62:45883 requested a full tablet report, sending...
I20260131 03:07:48.950778 22266 ts_manager.cc:194] Registered new tserver with Master: 2deb229679d44393a49c9c752d57affd (127.19.84.3:37585)
I20260131 03:07:48.951122 22266 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:56687
I20260131 03:07:48.960289 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260131 03:07:48.971086 22306 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:07:48.969999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43902 (local address 127.19.84.62:45883)
0131 03:07:48.970072 (+ 73us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:48.970075 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:48.970183 (+ 108us) server_negotiation.cc:408] Connection header received
0131 03:07:48.970287 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:48.970288 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:48.970306 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:48.970374 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:48.970992 (+ 618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43902: BlockingRecv error: recv got EOF from 127.0.0.1:43902 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260131 03:07:48.972138 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 22330
I20260131 03:07:48.976707 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 22466
I20260131 03:07:48.980767 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 22602
I20260131 03:07:48.985056 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 22234
2026-01-31T03:07:48Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2915 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:07:49 dist-test-slave-5nnp krb5kdc[22743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:07:49 dist-test-slave-5nnp krb5kdc[22743](info): set up 2 sockets
Jan 31 03:07:49 dist-test-slave-5nnp krb5kdc[22743](info): commencing operation
krb5kdc: starting...
W20260131 03:07:51.020574 19792 mini_kdc.cc:121] Time spent starting KDC: real 2.015s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:07:51 dist-test-slave-5nnp krb5kdc[22743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:07:51Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:07:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:51.163671 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:41801
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:46029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:41801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:51.243701 22759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:51.243908 22759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:51.243934 22759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:51.245482 22759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:07:51.245540 22759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:51.245555 22759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:51.245568 22759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:07:51.245580 22759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:07:51.247323 22759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:46029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:41801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:41801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:51.247622 22759 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:51.247844 22759 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:51.250581 22765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:51.250627 22759 server_base.cc:1060] running on GCE node
W20260131 03:07:51.250593 22767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:51.250583 22764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:51.251080 22759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:51.251338 22759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:51.252527 22759 hybrid_clock.cc:648] HybridClock initialized: now 1769828871252505 us; error 37 us; skew 500 ppm
Jan 31 03:07:51 dist-test-slave-5nnp krb5kdc[22743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828871, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:51.255021 22759 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:07:51.255688 22759 webserver.cc:492] Webserver started at http://127.19.84.62:45229/ using document root <none> and password file <none>
I20260131 03:07:51.255867 22759 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:51.255895 22759 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:51.255985 22759 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:51.256949 22759 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "cf629a64d48e42589ecf816e386b1ab2"
format_stamp: "Formatted at 2026-01-31 03:07:51 on dist-test-slave-5nnp"
server_key: "69f3249cc207636bd5ba3ade2e943347"
server_key_iv: "7b5f238bcb783a92b3b004e13e25bfcb"
server_key_version: "encryptionkey@0"
I20260131 03:07:51.257237 22759 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "cf629a64d48e42589ecf816e386b1ab2"
format_stamp: "Formatted at 2026-01-31 03:07:51 on dist-test-slave-5nnp"
server_key: "69f3249cc207636bd5ba3ade2e943347"
server_key_iv: "7b5f238bcb783a92b3b004e13e25bfcb"
server_key_version: "encryptionkey@0"
I20260131 03:07:51.258582 22759 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260131 03:07:51.259443 22774 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:51.259661 22759 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:51.259740 22759 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "cf629a64d48e42589ecf816e386b1ab2"
format_stamp: "Formatted at 2026-01-31 03:07:51 on dist-test-slave-5nnp"
server_key: "69f3249cc207636bd5ba3ade2e943347"
server_key_iv: "7b5f238bcb783a92b3b004e13e25bfcb"
server_key_version: "encryptionkey@0"
I20260131 03:07:51.259797 22759 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:51.268221 22759 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:51.273378 22759 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:51.273499 22759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:51.277814 22759 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:41801
I20260131 03:07:51.277904 22826 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:41801 every 8 connection(s)
I20260131 03:07:51.278218 22759 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:07:51.278988 22827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:51.281590 22827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf629a64d48e42589ecf816e386b1ab2: Bootstrap starting.
I20260131 03:07:51.282183 22827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf629a64d48e42589ecf816e386b1ab2: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:51.282445 22827 log.cc:826] T 00000000000000000000000000000000 P cf629a64d48e42589ecf816e386b1ab2: Log is configured to *not* fsync() on all Append() calls
I20260131 03:07:51.283226 22827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf629a64d48e42589ecf816e386b1ab2: No bootstrap required, opened a new log
I20260131 03:07:51.284493 22827 raft_consensus.cc:359] T 00000000000000000000000000000000 P cf629a64d48e42589ecf816e386b1ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf629a64d48e42589ecf816e386b1ab2" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41801 } }
I20260131 03:07:51.284641 22827 raft_consensus.cc:385] T 00000000000000000000000000000000 P cf629a64d48e42589ecf816e386b1ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:51.284684 22827 raft_consensus.cc:740] T 00000000000000000000000000000000 P cf629a64d48e42589ecf816e386b1ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf629a64d48e42589ecf816e386b1ab2, State: Initialized, Role: FOLLOWER
I20260131 03:07:51.284832 22827 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf629a64d48e42589ecf816e386b1ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf629a64d48e42589ecf816e386b1ab2" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41801 } }
I20260131 03:07:51.284919 22827 raft_consensus.cc:399] T 00000000000000000000000000000000 P cf629a64d48e42589ecf816e386b1ab2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:07:51.284957 22827 raft_consensus.cc:493] T 00000000000000000000000000000000 P cf629a64d48e42589ecf816e386b1ab2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:07:51.285008 22827 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cf629a64d48e42589ecf816e386b1ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:51.285552 22827 raft_consensus.cc:515] T 00000000000000000000000000000000 P cf629a64d48e42589ecf816e386b1ab2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf629a64d48e42589ecf816e386b1ab2" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41801 } }
I20260131 03:07:51.285724 22827 leader_election.cc:304] T 00000000000000000000000000000000 P cf629a64d48e42589ecf816e386b1ab2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf629a64d48e42589ecf816e386b1ab2; no voters:
I20260131 03:07:51.285923 22827 leader_election.cc:290] T 00000000000000000000000000000000 P cf629a64d48e42589ecf816e386b1ab2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:07:51.285961 22830 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cf629a64d48e42589ecf816e386b1ab2 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:07:51.286062 22830 raft_consensus.cc:697] T 00000000000000000000000000000000 P cf629a64d48e42589ecf816e386b1ab2 [term 1 LEADER]: Becoming Leader. State: Replica: cf629a64d48e42589ecf816e386b1ab2, State: Running, Role: LEADER
I20260131 03:07:51.286165 22827 sys_catalog.cc:565] T 00000000000000000000000000000000 P cf629a64d48e42589ecf816e386b1ab2 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:07:51.286140 22830 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf629a64d48e42589ecf816e386b1ab2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf629a64d48e42589ecf816e386b1ab2" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41801 } }
I20260131 03:07:51.286592 22832 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf629a64d48e42589ecf816e386b1ab2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf629a64d48e42589ecf816e386b1ab2. Latest consensus state: current_term: 1 leader_uuid: "cf629a64d48e42589ecf816e386b1ab2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf629a64d48e42589ecf816e386b1ab2" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41801 } } }
I20260131 03:07:51.286640 22831 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf629a64d48e42589ecf816e386b1ab2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf629a64d48e42589ecf816e386b1ab2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf629a64d48e42589ecf816e386b1ab2" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41801 } } }
I20260131 03:07:51.286756 22832 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf629a64d48e42589ecf816e386b1ab2 [sys.catalog]: This master's current role is: LEADER
I20260131 03:07:51.286880 22831 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf629a64d48e42589ecf816e386b1ab2 [sys.catalog]: This master's current role is: LEADER
I20260131 03:07:51.287166 22838 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:07:51.287710 22838 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:07:51.288014 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 22759
I20260131 03:07:51.288133 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:07:51.288306 19792 external_mini_cluster.cc:1468] Setting key 43d90eb6e82d4941ff9010f404be196d
I20260131 03:07:51.289664 22838 catalog_manager.cc:1357] Generated new cluster ID: 2f0f350a2ace40139a056b4a7d3c4b14
I20260131 03:07:51.289723 22838 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Jan 31 03:07:51 dist-test-slave-5nnp krb5kdc[22743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:51.298760 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:51.288773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55432 (local address 127.19.84.62:41801)
0131 03:07:51.289049 (+ 276us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:51.289058 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:51.289075 (+ 17us) server_negotiation.cc:408] Connection header received
0131 03:07:51.289162 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:51.289166 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:51.289202 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:51.289293 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:51.290317 (+ 1024us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:51.291510 (+ 1193us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:51.292311 (+ 801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:51.292539 (+ 228us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:51.295264 (+ 2725us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:51.295270 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:51.295270 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:51.295293 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:51.297353 (+ 2060us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:51.297720 (+ 367us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:51.297723 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:51.297723 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:51.297785 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:51.297952 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:51.297953 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:51.297954 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:51.298209 (+ 255us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:51.298367 (+ 158us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:51.298450 (+ 83us) server_negotiation.cc:300] Negotiation successful
0131 03:07:51.298577 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":94,"threads_started":1}
I20260131 03:07:51.310942 22838 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:07:51.311589 22838 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:07:51.320876 22838 catalog_manager.cc:6027] T 00000000000000000000000000000000 P cf629a64d48e42589ecf816e386b1ab2: Generated new TSK 0
I20260131 03:07:51.321103 22838 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:51.371516 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:41801
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:46029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:51.452222 22855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:51.452406 22855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:51.452430 22855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:51.453811 22855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:51.453953 22855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:51.454036 22855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:07:51.455641 22855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:46029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:41801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:51.455823 22855 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:51.456043 22855 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:51.458995 22863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:51.459059 22855 server_base.cc:1060] running on GCE node
W20260131 03:07:51.458995 22861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:51.459053 22860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:51.459415 22855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:51.459623 22855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:51.460780 22855 hybrid_clock.cc:648] HybridClock initialized: now 1769828871460761 us; error 39 us; skew 500 ppm
Jan 31 03:07:51 dist-test-slave-5nnp krb5kdc[22743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828871, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:51.463104 22855 init.cc:377] Logged in from keytab as kudu/127.19.84.1@KRBTEST.COM (short username kudu)
I20260131 03:07:51.463752 22855 webserver.cc:492] Webserver started at http://127.19.84.1:39361/ using document root <none> and password file <none>
I20260131 03:07:51.463965 22855 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:51.464023 22855 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:51.464159 22855 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:51.465137 22855 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "8b1b4b1873034d15a09598177068c840"
format_stamp: "Formatted at 2026-01-31 03:07:51 on dist-test-slave-5nnp"
server_key: "4b6cb92de54a9bda5c530e38ba898f30"
server_key_iv: "872e47b0facc73eee08eab1db1e2fc0d"
server_key_version: "encryptionkey@0"
I20260131 03:07:51.465471 22855 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "8b1b4b1873034d15a09598177068c840"
format_stamp: "Formatted at 2026-01-31 03:07:51 on dist-test-slave-5nnp"
server_key: "4b6cb92de54a9bda5c530e38ba898f30"
server_key_iv: "872e47b0facc73eee08eab1db1e2fc0d"
server_key_version: "encryptionkey@0"
I20260131 03:07:51.466800 22855 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:07:51.467686 22870 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:51.468041 22855 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:51.468145 22855 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "8b1b4b1873034d15a09598177068c840"
format_stamp: "Formatted at 2026-01-31 03:07:51 on dist-test-slave-5nnp"
server_key: "4b6cb92de54a9bda5c530e38ba898f30"
server_key_iv: "872e47b0facc73eee08eab1db1e2fc0d"
server_key_version: "encryptionkey@0"
I20260131 03:07:51.468222 22855 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:51.476390 22855 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:51.478470 22855 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:51.478636 22855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:51.478873 22855 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:07:51.479193 22855 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:07:51.479249 22855 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:51.479290 22855 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:07:51.479331 22855 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:51.485386 22855 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:38227
I20260131 03:07:51.485458 22983 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:38227 every 8 connection(s)
I20260131 03:07:51.485798 22855 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
I20260131 03:07:51.486936 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 22855
I20260131 03:07:51.487059 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:07:51.487260 19792 external_mini_cluster.cc:1468] Setting key 61469307cf60b1f07679241290a3a51a
Jan 31 03:07:51 dist-test-slave-5nnp krb5kdc[22743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828871, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.84.2@KRBTEST.COM; defaulting to no policy
I20260131 03:07:51.495828 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:51.486541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:45749 (local address 127.19.84.62:41801)
0131 03:07:51.486628 (+ 87us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:51.486630 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:51.486818 (+ 188us) server_negotiation.cc:408] Connection header received
0131 03:07:51.487528 (+ 710us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:51.487530 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:51.487551 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:51.487647 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:51.488560 (+ 913us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:51.489321 (+ 761us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:51.490168 (+ 847us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:51.490287 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:51.492991 (+ 2704us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:51.493002 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:51.493002 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:51.493009 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:51.494382 (+ 1373us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:51.494818 (+ 436us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:51.494819 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:51.494820 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:51.494852 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:51.495127 (+ 275us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:51.495129 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:51.495129 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:51.495236 (+ 107us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:51.495315 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:51.495622 (+ 307us) server_negotiation.cc:300] Negotiation successful
0131 03:07:51.495732 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260131 03:07:51.495955 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:51.486492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41801 (local address 127.19.84.1:45749)
0131 03:07:51.486677 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:51.486714 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:51.487420 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:51.487729 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:51.487733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:51.488183 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:51.488463 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:51.488466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:51.489428 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:51.489429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:51.490078 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:51.490081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:51.490228 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:51.491269 (+ 1041us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:51.491295 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:51.492889 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:51.494485 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:51.494492 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:51.494495 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:51.494728 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:51.494931 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:51.494932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:51.494932 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:51.495062 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:51.495303 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:51.495308 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:51.495487 (+ 179us) client_negotiation.cc:769] Sending connection context
0131 03:07:51.495603 (+ 116us) client_negotiation.cc:240] Negotiation successful
0131 03:07:51.495748 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":76,"threads_started":1}
I20260131 03:07:51.496446 22984 heartbeater.cc:344] Connected to a master server at 127.19.84.62:41801
I20260131 03:07:51.496573 22984 heartbeater.cc:461] Registering TS with master...
I20260131 03:07:51.496876 22984 heartbeater.cc:507] Master 127.19.84.62:41801 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.2@KRBTEST.COM" created.
I20260131 03:07:51.497403 22791 ts_manager.cc:194] Registered new tserver with Master: 8b1b4b1873034d15a09598177068c840 (127.19.84.1:38227)
I20260131 03:07:51.498065 22791 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.1@KRBTEST.COM'} at 127.19.84.1:45749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:51.533557 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:41801
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:46029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:51.610967 22991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:51.611142 22991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:51.611163 22991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:51.612517 22991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:51.612555 22991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:51.612581 22991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:07:51.614054 22991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:46029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:41801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:51.614231 22991 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:51.614362 22991 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:51.616760 22997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:51.616953 22991 server_base.cc:1060] running on GCE node
W20260131 03:07:51.616814 22999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:51.616750 22996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:51.617223 22991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:51.617436 22991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:51.618579 22991 hybrid_clock.cc:648] HybridClock initialized: now 1769828871618564 us; error 32 us; skew 500 ppm
Jan 31 03:07:51 dist-test-slave-5nnp krb5kdc[22743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828871, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:51.620918 22991 init.cc:377] Logged in from keytab as kudu/127.19.84.2@KRBTEST.COM (short username kudu)
I20260131 03:07:51.621419 22991 webserver.cc:492] Webserver started at http://127.19.84.2:34531/ using document root <none> and password file <none>
I20260131 03:07:51.621616 22991 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:51.621657 22991 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:51.621783 22991 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:51.622702 22991 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "f4fe0b2e2d8143f98cbe03ba2d488d3d"
format_stamp: "Formatted at 2026-01-31 03:07:51 on dist-test-slave-5nnp"
server_key: "f8c6b1ed01a6690de0ce1364fa6b0935"
server_key_iv: "f86392f24e4624f9af44bd0ed0ee16d3"
server_key_version: "encryptionkey@0"
I20260131 03:07:51.623019 22991 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "f4fe0b2e2d8143f98cbe03ba2d488d3d"
format_stamp: "Formatted at 2026-01-31 03:07:51 on dist-test-slave-5nnp"
server_key: "f8c6b1ed01a6690de0ce1364fa6b0935"
server_key_iv: "f86392f24e4624f9af44bd0ed0ee16d3"
server_key_version: "encryptionkey@0"
I20260131 03:07:51.624258 22991 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260131 03:07:51.624928 23006 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:51.625100 22991 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260131 03:07:51.625186 22991 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "f4fe0b2e2d8143f98cbe03ba2d488d3d"
format_stamp: "Formatted at 2026-01-31 03:07:51 on dist-test-slave-5nnp"
server_key: "f8c6b1ed01a6690de0ce1364fa6b0935"
server_key_iv: "f86392f24e4624f9af44bd0ed0ee16d3"
server_key_version: "encryptionkey@0"
I20260131 03:07:51.625262 22991 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:51.661592 22991 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:51.663762 22991 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:51.663938 22991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:51.664196 22991 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:07:51.664499 22991 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:07:51.664552 22991 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:51.664623 22991 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:07:51.664654 22991 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:51.670785 22991 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:41097
I20260131 03:07:51.670825 23119 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:41097 every 8 connection(s)
I20260131 03:07:51.671165 22991 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
Jan 31 03:07:51 dist-test-slave-5nnp krb5kdc[22743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828871, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:51.678218 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 22991
I20260131 03:07:51.678328 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:07:51.678507 19792 external_mini_cluster.cc:1468] Setting key d2ec9bc72b8c4327cae4394ed041231f
I20260131 03:07:51.680032 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:51.671833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41801 (local address 127.19.84.2:56793)
0131 03:07:51.671995 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:51.672023 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:51.672476 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:51.672767 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:51.672770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:51.673084 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:51.673362 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:51.673365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:51.674016 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:51.674017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:51.674389 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:51.674391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:51.674496 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:51.675346 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:51.675360 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:51.676770 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:51.678487 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:51.678493 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:51.678495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:51.678814 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:51.678992 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:51.678992 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:51.678993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:51.679095 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:51.679347 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:51.679352 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:51.679661 (+ 309us) client_negotiation.cc:769] Sending connection context
0131 03:07:51.679768 (+ 107us) client_negotiation.cc:240] Negotiation successful
0131 03:07:51.679879 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
I20260131 03:07:51.680060 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:51.671890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:56793 (local address 127.19.84.62:41801)
0131 03:07:51.671969 (+ 79us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:51.671972 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:51.672103 (+ 131us) server_negotiation.cc:408] Connection header received
0131 03:07:51.672604 (+ 501us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:51.672605 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:51.672652 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:51.672718 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:51.673437 (+ 719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:51.673937 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:51.674455 (+ 518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:51.674565 (+ 110us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:51.676851 (+ 2286us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:51.676856 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:51.676856 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:51.676863 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:51.678407 (+ 1544us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:51.678889 (+ 482us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:51.678891 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:51.678891 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:51.678923 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:51.679157 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:51.679157 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:51.679157 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:51.679286 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:51.679352 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:51.679818 (+ 466us) server_negotiation.cc:300] Negotiation successful
0131 03:07:51.679922 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260131 03:07:51.680653 23120 heartbeater.cc:344] Connected to a master server at 127.19.84.62:41801
I20260131 03:07:51.680785 23120 heartbeater.cc:461] Registering TS with master...
I20260131 03:07:51.681010 23120 heartbeater.cc:507] Master 127.19.84.62:41801 requested a full tablet report, sending...
I20260131 03:07:51.681433 22791 ts_manager.cc:194] Registered new tserver with Master: f4fe0b2e2d8143f98cbe03ba2d488d3d (127.19.84.2:41097)
I20260131 03:07:51.681851 22791 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.2@KRBTEST.COM'} at 127.19.84.2:56793
WARNING: no policy specified for kudu/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:51.724454 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:41801
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:46029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:51.806977 23127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:51.807199 23127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:51.807225 23127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:51.808936 23127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:51.809000 23127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:51.809029 23127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:07:51.810703 23127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:46029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:41801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:51.810921 23127 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:51.811072 23127 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:51.814080 23133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:51.814079 23132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:51.814107 23127 server_base.cc:1060] running on GCE node
W20260131 03:07:51.814079 23135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:51.814615 23127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:51.814877 23127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:51.816054 23127 hybrid_clock.cc:648] HybridClock initialized: now 1769828871816039 us; error 26 us; skew 500 ppm
Jan 31 03:07:51 dist-test-slave-5nnp krb5kdc[22743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828871, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:51.818686 23127 init.cc:377] Logged in from keytab as kudu/127.19.84.3@KRBTEST.COM (short username kudu)
I20260131 03:07:51.819283 23127 webserver.cc:492] Webserver started at http://127.19.84.3:36545/ using document root <none> and password file <none>
I20260131 03:07:51.819523 23127 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:51.819581 23127 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:51.819718 23127 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:51.820679 23127 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "582596f8b6c8412ca08885cbca3173a8"
format_stamp: "Formatted at 2026-01-31 03:07:51 on dist-test-slave-5nnp"
server_key: "96aea477c134f8b8a7ad7829579d8a14"
server_key_iv: "d73a40e79ba63797748ed64278557dac"
server_key_version: "encryptionkey@0"
I20260131 03:07:51.821017 23127 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "582596f8b6c8412ca08885cbca3173a8"
format_stamp: "Formatted at 2026-01-31 03:07:51 on dist-test-slave-5nnp"
server_key: "96aea477c134f8b8a7ad7829579d8a14"
server_key_iv: "d73a40e79ba63797748ed64278557dac"
server_key_version: "encryptionkey@0"
I20260131 03:07:51.822283 23127 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:07:51.823105 23142 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:51.823323 23127 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:51.823439 23127 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "582596f8b6c8412ca08885cbca3173a8"
format_stamp: "Formatted at 2026-01-31 03:07:51 on dist-test-slave-5nnp"
server_key: "96aea477c134f8b8a7ad7829579d8a14"
server_key_iv: "d73a40e79ba63797748ed64278557dac"
server_key_version: "encryptionkey@0"
I20260131 03:07:51.823521 23127 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:51.831918 23127 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:51.834002 23127 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:51.834162 23127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:51.834396 23127 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:07:51.834717 23127 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:07:51.834772 23127 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:51.834817 23127 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:07:51.835014 23127 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:51.842661 23127 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:39925
I20260131 03:07:51.842690 23255 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:39925 every 8 connection(s)
I20260131 03:07:51.843080 23127 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
Jan 31 03:07:51 dist-test-slave-5nnp krb5kdc[22743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828871, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:51.849129 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 23127
I20260131 03:07:51.849226 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:07:51.849404 19792 external_mini_cluster.cc:1468] Setting key bc848e5deb1ed2928d8752037db7a03e
I20260131 03:07:51.853137 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:51.843974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:44879 (local address 127.19.84.62:41801)
0131 03:07:51.844044 (+ 70us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:51.844046 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:51.844058 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:07:51.844576 (+ 518us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:51.844578 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:51.844589 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:51.844698 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:51.845591 (+ 893us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:51.846093 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:51.846722 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:51.846857 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:51.849708 (+ 2851us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:51.849712 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:51.849713 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:51.849723 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:51.851600 (+ 1877us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:51.852068 (+ 468us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:51.852069 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:51.852069 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:51.852095 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:51.852456 (+ 361us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:51.852457 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:51.852457 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:51.852577 (+ 120us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:51.852671 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:51.852971 (+ 300us) server_negotiation.cc:300] Negotiation successful
0131 03:07:51.853062 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260131 03:07:51.853320 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:51.843796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41801 (local address 127.19.84.3:44879)
0131 03:07:51.843954 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:51.843988 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:51.844470 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:51.844791 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:51.844795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:51.845160 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:51.845445 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:51.845448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:51.846189 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:51.846191 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:51.846597 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:51.846598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:51.846726 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:51.847784 (+ 1058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:51.847804 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:51.849585 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:51.851712 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:51.851719 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:51.851722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:51.851969 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:51.852226 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:51.852227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:51.852228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:51.852335 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:51.852683 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:51.852689 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:51.852830 (+ 141us) client_negotiation.cc:769] Sending connection context
0131 03:07:51.852942 (+ 112us) client_negotiation.cc:240] Negotiation successful
0131 03:07:51.853072 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20260131 03:07:51.853891 23256 heartbeater.cc:344] Connected to a master server at 127.19.84.62:41801
I20260131 03:07:51.854039 23256 heartbeater.cc:461] Registering TS with master...
I20260131 03:07:51.854282 23256 heartbeater.cc:507] Master 127.19.84.62:41801 requested a full tablet report, sending...
I20260131 03:07:51.854871 22791 ts_manager.cc:194] Registered new tserver with Master: 582596f8b6c8412ca08885cbca3173a8 (127.19.84.3:39925)
I20260131 03:07:51.855464 22791 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:44879
I20260131 03:07:51.862082 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260131 03:07:51.959264 22849 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:07:51.957407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55438 (local address 127.19.84.62:41801)
0131 03:07:51.957473 (+ 66us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:51.957475 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:51.957672 (+ 197us) server_negotiation.cc:408] Connection header received
0131 03:07:51.958231 (+ 559us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:51.958232 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:51.958242 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:51.958307 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:51.959184 (+ 877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55438: BlockingRecv error: recv got EOF from 127.0.0.1:55438 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260131 03:07:51.960054 22849 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:07:51.959425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55448 (local address 127.19.84.62:41801)
0131 03:07:51.959459 (+ 34us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:51.959461 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:51.959629 (+ 168us) server_negotiation.cc:408] Connection header received
0131 03:07:51.959654 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:51.959655 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:51.959663 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:51.959712 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:51.959993 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55448: BlockingRecv error: recv got EOF from 127.0.0.1:55448 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260131 03:07:51.960728 22849 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:07:51.960186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55458 (local address 127.19.84.62:41801)
0131 03:07:51.960231 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:51.960233 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:51.960369 (+ 136us) server_negotiation.cc:408] Connection header received
0131 03:07:51.960392 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:51.960393 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:51.960411 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:51.960457 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:51.960664 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55458: BlockingRecv error: recv got EOF from 127.0.0.1:55458 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260131 03:07:51.962440 22849 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:07:51.961923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.19.84.62:41801)
0131 03:07:51.961982 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:51.961984 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:51.961993 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:07:51.962052 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:51.962053 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:51.962062 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:51.962126 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:51.962412 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55470: BlockingRecv error: recv got EOF from 127.0.0.1:55470 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260131 03:07:51.968657 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 22855
I20260131 03:07:51.973441 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 22991
I20260131 03:07:51.978049 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 23127
I20260131 03:07:51.982343 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 22759
2026-01-31T03:07:51Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (2997 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:07:52 dist-test-slave-5nnp krb5kdc[23279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:07:52 dist-test-slave-5nnp krb5kdc[23279](info): set up 2 sockets
Jan 31 03:07:52 dist-test-slave-5nnp krb5kdc[23279](info): commencing operation
krb5kdc: starting...
W20260131 03:07:54.103165 19792 mini_kdc.cc:121] Time spent starting KDC: real 2.100s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260131 03:07:54.418030 19792 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.315s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260131 03:07:54.607093 19792 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.189s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260131 03:07:54.888464 19792 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.281s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/krb5kdc/test-user.keytab.
W20260131 03:07:55.722239 19792 mini_kdc.cc:290] Time spent creating keytab for test-user: real 0.834s user 0.000s sys 0.002s
Jan 31 03:07:55 dist-test-slave-5nnp krb5kdc[23279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:07:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:07:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/krb5kdc/kudu.keytab.
W20260131 03:07:56.328552 19792 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.19.84.62: real 0.585s user 0.000s sys 0.003s
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:56.355118 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:36575
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:42285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:36575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:56.443614 23295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:56.443840 23295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:56.443882 23295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:56.445430 23295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:07:56.445508 23295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:56.445533 23295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:56.445577 23295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:07:56.445602 23295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:07:56.447326 23295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:42285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:36575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:36575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:56.447671 23295 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:56.447844 23295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:56.450939 23301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:56.450690 23303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:56.451082 23295 server_base.cc:1060] running on GCE node
W20260131 03:07:56.450860 23300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:56.451649 23295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:56.451936 23295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:56.453101 23295 hybrid_clock.cc:648] HybridClock initialized: now 1769828876453078 us; error 39 us; skew 500 ppm
Jan 31 03:07:56 dist-test-slave-5nnp krb5kdc[23279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828876, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:56.456146 23295 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:07:56.456815 23295 webserver.cc:492] Webserver started at http://127.19.84.62:41343/ using document root <none> and password file <none>
I20260131 03:07:56.457042 23295 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:56.457082 23295 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:56.457201 23295 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:56.458459 23295 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "50988d5b6e7f47178f42ae15c967ac5a"
format_stamp: "Formatted at 2026-01-31 03:07:56 on dist-test-slave-5nnp"
server_key: "7fa41d94aef0b14743538515e5d901bd"
server_key_iv: "f796ce2f9db6e1a3775263ba1ef4d919"
server_key_version: "encryptionkey@0"
I20260131 03:07:56.458878 23295 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "50988d5b6e7f47178f42ae15c967ac5a"
format_stamp: "Formatted at 2026-01-31 03:07:56 on dist-test-slave-5nnp"
server_key: "7fa41d94aef0b14743538515e5d901bd"
server_key_iv: "f796ce2f9db6e1a3775263ba1ef4d919"
server_key_version: "encryptionkey@0"
I20260131 03:07:56.460403 23295 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:07:56.461319 23310 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:56.461647 23295 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:56.461735 23295 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "50988d5b6e7f47178f42ae15c967ac5a"
format_stamp: "Formatted at 2026-01-31 03:07:56 on dist-test-slave-5nnp"
server_key: "7fa41d94aef0b14743538515e5d901bd"
server_key_iv: "f796ce2f9db6e1a3775263ba1ef4d919"
server_key_version: "encryptionkey@0"
I20260131 03:07:56.461846 23295 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:56.472952 23295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:56.478921 23295 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:56.479094 23295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:56.483254 23295 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:36575
I20260131 03:07:56.483302 23362 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:36575 every 8 connection(s)
I20260131 03:07:56.483683 23295 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:07:56.484380 23363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:56.486989 23363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50988d5b6e7f47178f42ae15c967ac5a: Bootstrap starting.
I20260131 03:07:56.487669 23363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50988d5b6e7f47178f42ae15c967ac5a: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:56.487890 23363 log.cc:826] T 00000000000000000000000000000000 P 50988d5b6e7f47178f42ae15c967ac5a: Log is configured to *not* fsync() on all Append() calls
I20260131 03:07:56.488790 23363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50988d5b6e7f47178f42ae15c967ac5a: No bootstrap required, opened a new log
I20260131 03:07:56.490000 23363 raft_consensus.cc:359] T 00000000000000000000000000000000 P 50988d5b6e7f47178f42ae15c967ac5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50988d5b6e7f47178f42ae15c967ac5a" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36575 } }
I20260131 03:07:56.490126 23363 raft_consensus.cc:385] T 00000000000000000000000000000000 P 50988d5b6e7f47178f42ae15c967ac5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:56.490147 23363 raft_consensus.cc:740] T 00000000000000000000000000000000 P 50988d5b6e7f47178f42ae15c967ac5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50988d5b6e7f47178f42ae15c967ac5a, State: Initialized, Role: FOLLOWER
I20260131 03:07:56.490289 23363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50988d5b6e7f47178f42ae15c967ac5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50988d5b6e7f47178f42ae15c967ac5a" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36575 } }
I20260131 03:07:56.490388 23363 raft_consensus.cc:399] T 00000000000000000000000000000000 P 50988d5b6e7f47178f42ae15c967ac5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:07:56.490430 23363 raft_consensus.cc:493] T 00000000000000000000000000000000 P 50988d5b6e7f47178f42ae15c967ac5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:07:56.490391 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 23295
I20260131 03:07:56.490502 23363 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 50988d5b6e7f47178f42ae15c967ac5a [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:56.490506 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:07:56.490787 19792 external_mini_cluster.cc:1468] Setting key 558e37be84da9b6d6979af3fcff32b97
I20260131 03:07:56.491462 23363 raft_consensus.cc:515] T 00000000000000000000000000000000 P 50988d5b6e7f47178f42ae15c967ac5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50988d5b6e7f47178f42ae15c967ac5a" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36575 } }
I20260131 03:07:56.491603 23363 leader_election.cc:304] T 00000000000000000000000000000000 P 50988d5b6e7f47178f42ae15c967ac5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50988d5b6e7f47178f42ae15c967ac5a; no voters:
I20260131 03:07:56.491768 23363 leader_election.cc:290] T 00000000000000000000000000000000 P 50988d5b6e7f47178f42ae15c967ac5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:07:56.491827 23368 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 50988d5b6e7f47178f42ae15c967ac5a [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:07:56.492000 23368 raft_consensus.cc:697] T 00000000000000000000000000000000 P 50988d5b6e7f47178f42ae15c967ac5a [term 1 LEADER]: Becoming Leader. State: Replica: 50988d5b6e7f47178f42ae15c967ac5a, State: Running, Role: LEADER
I20260131 03:07:56.492215 23368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50988d5b6e7f47178f42ae15c967ac5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50988d5b6e7f47178f42ae15c967ac5a" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36575 } }
I20260131 03:07:56.492046 23363 sys_catalog.cc:565] T 00000000000000000000000000000000 P 50988d5b6e7f47178f42ae15c967ac5a [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:07:56.492671 23369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50988d5b6e7f47178f42ae15c967ac5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50988d5b6e7f47178f42ae15c967ac5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50988d5b6e7f47178f42ae15c967ac5a" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36575 } } }
I20260131 03:07:56.492756 23369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50988d5b6e7f47178f42ae15c967ac5a [sys.catalog]: This master's current role is: LEADER
I20260131 03:07:56.493229 23370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50988d5b6e7f47178f42ae15c967ac5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50988d5b6e7f47178f42ae15c967ac5a. Latest consensus state: current_term: 1 leader_uuid: "50988d5b6e7f47178f42ae15c967ac5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50988d5b6e7f47178f42ae15c967ac5a" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36575 } } }
I20260131 03:07:56.493819 23370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50988d5b6e7f47178f42ae15c967ac5a [sys.catalog]: This master's current role is: LEADER
I20260131 03:07:56.494043 23372 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:07:56.494530 23372 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:07:56.496454 23372 catalog_manager.cc:1357] Generated new cluster ID: ffddda89319d416b8f52d4fe94540963
I20260131 03:07:56.496536 23372 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Jan 31 03:07:56 dist-test-slave-5nnp krb5kdc[23279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:56.502187 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:56.491420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33024 (local address 127.19.84.62:36575)
0131 03:07:56.491645 (+ 225us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:56.491664 (+ 19us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:56.491694 (+ 30us) server_negotiation.cc:408] Connection header received
0131 03:07:56.491872 (+ 178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:56.491879 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:56.491938 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:56.493132 (+ 1194us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:56.493268 (+ 136us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:56.494603 (+ 1335us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:56.495648 (+ 1045us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:56.495928 (+ 280us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:56.498358 (+ 2430us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:56.498383 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:56.498383 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:56.498410 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:56.500609 (+ 2199us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:56.501025 (+ 416us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:56.501028 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:56.501029 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:56.501116 (+ 87us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:56.501294 (+ 178us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:56.501295 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:56.501295 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:56.501589 (+ 294us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:56.501752 (+ 163us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:56.501842 (+ 90us) server_negotiation.cc:300] Negotiation successful
0131 03:07:56.501981 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":118,"threads_started":1}
I20260131 03:07:56.503209 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:07:56.510721 23372 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:07:56.511299 23372 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:07:56.514683 23372 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 50988d5b6e7f47178f42ae15c967ac5a: Generated new TSK 0
I20260131 03:07:56.514900 23372 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:07:56.519470 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:56.515855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33032 (local address 127.19.84.62:36575)
0131 03:07:56.515920 (+ 65us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:56.515923 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:56.515936 (+ 13us) server_negotiation.cc:408] Connection header received
0131 03:07:56.515961 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:56.515961 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:56.515970 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:56.516036 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:56.516781 (+ 745us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:56.517374 (+ 593us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:56.518255 (+ 881us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:56.518440 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:56.518903 (+ 463us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:56.518908 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:56.518908 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:07:56.519124 (+ 216us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:56.519143 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:07:56.519148 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:56.519205 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:56.519297 (+ 92us) server_negotiation.cc:300] Negotiation successful
0131 03:07:56.519329 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260131 03:07:56.522212 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 23295
2026-01-31T03:07:56Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (4537 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:07:56 dist-test-slave-5nnp krb5kdc[23396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:07:56 dist-test-slave-5nnp krb5kdc[23396](info): set up 2 sockets
Jan 31 03:07:56 dist-test-slave-5nnp krb5kdc[23396](info): commencing operation
krb5kdc: starting...
W20260131 03:07:58.560873 19792 mini_kdc.cc:121] Time spent starting KDC: real 2.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:07:58 dist-test-slave-5nnp krb5kdc[23396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:07:58Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:07:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:58.713019 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:39853
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:39853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:58.800628 23412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:58.800909 23412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:58.800956 23412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:58.802578 23412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:07:58.802680 23412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:58.802704 23412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:58.802724 23412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:07:58.802754 23412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:07:58.804709 23412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:39853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:39853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:58.805019 23412 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:58.805230 23412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:58.808698 23418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:58.808704 23417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:58.808954 23420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:58.809597 23412 server_base.cc:1060] running on GCE node
I20260131 03:07:58.809873 23412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:58.810453 23412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:58.811688 23412 hybrid_clock.cc:648] HybridClock initialized: now 1769828878811685 us; error 95 us; skew 500 ppm
Jan 31 03:07:58 dist-test-slave-5nnp krb5kdc[23396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828878, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:58.814646 23412 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:07:58.815310 23412 webserver.cc:492] Webserver started at http://127.19.84.62:40151/ using document root <none> and password file <none>
I20260131 03:07:58.815557 23412 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:58.815598 23412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:58.815724 23412 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:58.816733 23412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "cd9590cd78fe4ff2b1cedd23db018e0b"
format_stamp: "Formatted at 2026-01-31 03:07:58 on dist-test-slave-5nnp"
server_key: "c475c1971e724b88bc69a99355ac585f"
server_key_iv: "d8f0a9db229bda33c7c811235fb31de7"
server_key_version: "encryptionkey@0"
I20260131 03:07:58.817085 23412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "cd9590cd78fe4ff2b1cedd23db018e0b"
format_stamp: "Formatted at 2026-01-31 03:07:58 on dist-test-slave-5nnp"
server_key: "c475c1971e724b88bc69a99355ac585f"
server_key_iv: "d8f0a9db229bda33c7c811235fb31de7"
server_key_version: "encryptionkey@0"
I20260131 03:07:58.818475 23412 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:07:58.819314 23427 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:58.819612 23412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:07:58.819686 23412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "cd9590cd78fe4ff2b1cedd23db018e0b"
format_stamp: "Formatted at 2026-01-31 03:07:58 on dist-test-slave-5nnp"
server_key: "c475c1971e724b88bc69a99355ac585f"
server_key_iv: "d8f0a9db229bda33c7c811235fb31de7"
server_key_version: "encryptionkey@0"
I20260131 03:07:58.819742 23412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:58.834395 23412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:58.836897 23412 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:58.837023 23412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:58.840983 23412 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:39853
I20260131 03:07:58.841073 23479 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:39853 every 8 connection(s)
I20260131 03:07:58.841395 23412 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:07:58.842149 23480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:07:58.845778 23480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd9590cd78fe4ff2b1cedd23db018e0b: Bootstrap starting.
I20260131 03:07:58.846449 23480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd9590cd78fe4ff2b1cedd23db018e0b: Neither blocks nor log segments found. Creating new log.
I20260131 03:07:58.846848 23480 log.cc:826] T 00000000000000000000000000000000 P cd9590cd78fe4ff2b1cedd23db018e0b: Log is configured to *not* fsync() on all Append() calls
I20260131 03:07:58.847988 23480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd9590cd78fe4ff2b1cedd23db018e0b: No bootstrap required, opened a new log
I20260131 03:07:58.848443 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 23412
I20260131 03:07:58.848620 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:07:58.848811 19792 external_mini_cluster.cc:1468] Setting key ee5febbd345861a2964383b97f867275
I20260131 03:07:58.850095 23480 raft_consensus.cc:359] T 00000000000000000000000000000000 P cd9590cd78fe4ff2b1cedd23db018e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9590cd78fe4ff2b1cedd23db018e0b" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 39853 } }
I20260131 03:07:58.850287 23480 raft_consensus.cc:385] T 00000000000000000000000000000000 P cd9590cd78fe4ff2b1cedd23db018e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:07:58.850332 23480 raft_consensus.cc:740] T 00000000000000000000000000000000 P cd9590cd78fe4ff2b1cedd23db018e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd9590cd78fe4ff2b1cedd23db018e0b, State: Initialized, Role: FOLLOWER
I20260131 03:07:58.850556 23480 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd9590cd78fe4ff2b1cedd23db018e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9590cd78fe4ff2b1cedd23db018e0b" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 39853 } }
I20260131 03:07:58.850782 23480 raft_consensus.cc:399] T 00000000000000000000000000000000 P cd9590cd78fe4ff2b1cedd23db018e0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:07:58.850843 23480 raft_consensus.cc:493] T 00000000000000000000000000000000 P cd9590cd78fe4ff2b1cedd23db018e0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:07:58.850900 23480 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cd9590cd78fe4ff2b1cedd23db018e0b [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:07:58.851557 23480 raft_consensus.cc:515] T 00000000000000000000000000000000 P cd9590cd78fe4ff2b1cedd23db018e0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9590cd78fe4ff2b1cedd23db018e0b" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 39853 } }
I20260131 03:07:58.851701 23480 leader_election.cc:304] T 00000000000000000000000000000000 P cd9590cd78fe4ff2b1cedd23db018e0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd9590cd78fe4ff2b1cedd23db018e0b; no voters:
I20260131 03:07:58.851907 23480 leader_election.cc:290] T 00000000000000000000000000000000 P cd9590cd78fe4ff2b1cedd23db018e0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:07:58.851940 23485 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cd9590cd78fe4ff2b1cedd23db018e0b [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:07:58.852049 23485 raft_consensus.cc:697] T 00000000000000000000000000000000 P cd9590cd78fe4ff2b1cedd23db018e0b [term 1 LEADER]: Becoming Leader. State: Replica: cd9590cd78fe4ff2b1cedd23db018e0b, State: Running, Role: LEADER
I20260131 03:07:58.852186 23480 sys_catalog.cc:565] T 00000000000000000000000000000000 P cd9590cd78fe4ff2b1cedd23db018e0b [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:07:58.852231 23485 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd9590cd78fe4ff2b1cedd23db018e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9590cd78fe4ff2b1cedd23db018e0b" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 39853 } }
I20260131 03:07:58.852674 23486 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd9590cd78fe4ff2b1cedd23db018e0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd9590cd78fe4ff2b1cedd23db018e0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9590cd78fe4ff2b1cedd23db018e0b" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 39853 } } }
I20260131 03:07:58.852762 23486 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd9590cd78fe4ff2b1cedd23db018e0b [sys.catalog]: This master's current role is: LEADER
I20260131 03:07:58.852969 23487 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd9590cd78fe4ff2b1cedd23db018e0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd9590cd78fe4ff2b1cedd23db018e0b. Latest consensus state: current_term: 1 leader_uuid: "cd9590cd78fe4ff2b1cedd23db018e0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9590cd78fe4ff2b1cedd23db018e0b" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 39853 } } }
I20260131 03:07:58.853264 23487 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd9590cd78fe4ff2b1cedd23db018e0b [sys.catalog]: This master's current role is: LEADER
W20260131 03:07:58.854470 23501 catalog_manager.cc:1568] T 00000000000000000000000000000000 P cd9590cd78fe4ff2b1cedd23db018e0b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:07:58.854543 23501 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:07:58.854615 23491 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:07:58.854815 23491 catalog_manager.cc:1494] Initializing Kudu cluster ID...
Jan 31 03:07:58 dist-test-slave-5nnp krb5kdc[23396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:58.856662 23491 catalog_manager.cc:1357] Generated new cluster ID: cbc67863bdc24e12a2e67d01366f1baf
I20260131 03:07:58.856716 23491 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:07:58.859550 23484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:58.849459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40338 (local address 127.19.84.62:39853)
0131 03:07:58.849656 (+ 197us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:58.849666 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:58.849688 (+ 22us) server_negotiation.cc:408] Connection header received
0131 03:07:58.850158 (+ 470us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:58.850172 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:58.850241 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:58.850411 (+ 170us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:58.851368 (+ 957us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:58.852463 (+ 1095us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:58.853552 (+ 1089us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:58.853810 (+ 258us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:58.856215 (+ 2405us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:58.856226 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:58.856226 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:58.856255 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:58.858106 (+ 1851us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:58.858524 (+ 418us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:58.858528 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:58.858528 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:58.858584 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:58.858755 (+ 171us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:58.858756 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:58.858756 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:58.858967 (+ 211us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:58.859119 (+ 152us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:58.859232 (+ 113us) server_negotiation.cc:300] Negotiation successful
0131 03:07:58.859365 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":97,"threads_started":1}
I20260131 03:07:58.874806 23491 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:07:58.875626 23491 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:07:58.885268 23491 catalog_manager.cc:6027] T 00000000000000000000000000000000 P cd9590cd78fe4ff2b1cedd23db018e0b: Generated new TSK 0
I20260131 03:07:58.885504 23491 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:58.940537 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:39853
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:59.026180 23508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:59.026394 23508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:59.026420 23508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:59.028228 23508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:59.028311 23508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:59.028354 23508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:07:59.030458 23508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:39853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:59.030716 23508 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:59.030884 23508 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260131 03:07:59.034142 23508 server_base.cc:1060] running on GCE node
W20260131 03:07:59.034114 23513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:59.034112 23514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:59.034122 23516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:59.034819 23508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:59.035081 23508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:59.036244 23508 hybrid_clock.cc:648] HybridClock initialized: now 1769828879036200 us; error 96 us; skew 500 ppm
Jan 31 03:07:59 dist-test-slave-5nnp krb5kdc[23396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828879, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:59.038993 23508 init.cc:377] Logged in from keytab as kudu/127.19.84.1@KRBTEST.COM (short username kudu)
I20260131 03:07:59.039712 23508 webserver.cc:492] Webserver started at http://127.19.84.1:46745/ using document root <none> and password file <none>
I20260131 03:07:59.039970 23508 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:59.040010 23508 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:59.040139 23508 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:59.041154 23508 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "a5b0336c30f648cc9b62d292b6c18368"
format_stamp: "Formatted at 2026-01-31 03:07:59 on dist-test-slave-5nnp"
server_key: "f1a1c3d7eaa527198874aa8593b2039c"
server_key_iv: "8429225fbe4e7faf5a88221db14f1e99"
server_key_version: "encryptionkey@0"
I20260131 03:07:59.041486 23508 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "a5b0336c30f648cc9b62d292b6c18368"
format_stamp: "Formatted at 2026-01-31 03:07:59 on dist-test-slave-5nnp"
server_key: "f1a1c3d7eaa527198874aa8593b2039c"
server_key_iv: "8429225fbe4e7faf5a88221db14f1e99"
server_key_version: "encryptionkey@0"
I20260131 03:07:59.042866 23508 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260131 03:07:59.043644 23523 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:59.043835 23508 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:07:59.043931 23508 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "a5b0336c30f648cc9b62d292b6c18368"
format_stamp: "Formatted at 2026-01-31 03:07:59 on dist-test-slave-5nnp"
server_key: "f1a1c3d7eaa527198874aa8593b2039c"
server_key_iv: "8429225fbe4e7faf5a88221db14f1e99"
server_key_version: "encryptionkey@0"
I20260131 03:07:59.044015 23508 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:59.052446 23508 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:59.055552 23508 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:59.055737 23508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:59.055985 23508 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:07:59.056318 23508 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:07:59.056352 23508 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:59.056399 23508 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:07:59.056425 23508 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:59.062551 23508 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:40439
I20260131 03:07:59.062661 23636 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:40439 every 8 connection(s)
I20260131 03:07:59.062986 23508 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
I20260131 03:07:59.065306 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 23508
I20260131 03:07:59.065440 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:07:59.065630 19792 external_mini_cluster.cc:1468] Setting key db8be9fdc08f0d33a25e80afb99829b6
Jan 31 03:07:59 dist-test-slave-5nnp krb5kdc[23396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828879, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:59.073627 23484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:59.063794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:33491 (local address 127.19.84.62:39853)
0131 03:07:59.063874 (+ 80us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:59.063877 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:59.064043 (+ 166us) server_negotiation.cc:408] Connection header received
0131 03:07:59.064567 (+ 524us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:59.064569 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:59.064592 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:59.064670 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:59.065642 (+ 972us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:59.066298 (+ 656us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:59.067172 (+ 874us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:59.067316 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:59.070115 (+ 2799us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:59.070121 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:59.070121 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:59.070130 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:59.072020 (+ 1890us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:59.072460 (+ 440us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:59.072462 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:59.072462 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:59.072494 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:59.072729 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:59.072730 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:59.072730 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:59.072978 (+ 248us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:59.073068 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:59.073388 (+ 320us) server_negotiation.cc:300] Negotiation successful
0131 03:07:59.073506 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260131 03:07:59.073640 23639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:59.063757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:39853 (local address 127.19.84.1:33491)
0131 03:07:59.063933 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:59.063961 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:59.064461 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:59.064738 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:59.064742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:59.065064 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:59.065513 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:59.065518 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:59.066386 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:59.066388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:59.066993 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:59.066997 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:59.067152 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:59.068233 (+ 1081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:59.068252 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:59.070011 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:59.072122 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:59.072128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:59.072131 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:59.072373 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:59.072566 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:59.072566 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:59.072567 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:59.072660 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:59.073093 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:59.073098 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:59.073234 (+ 136us) client_negotiation.cc:769] Sending connection context
0131 03:07:59.073336 (+ 102us) client_negotiation.cc:240] Negotiation successful
0131 03:07:59.073459 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260131 03:07:59.074445 23637 heartbeater.cc:344] Connected to a master server at 127.19.84.62:39853
I20260131 03:07:59.074558 23637 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.19.84.2@KRBTEST.COM; defaulting to no policy
I20260131 03:07:59.074806 23637 heartbeater.cc:507] Master 127.19.84.62:39853 requested a full tablet report, sending...
I20260131 03:07:59.075476 23444 ts_manager.cc:194] Registered new tserver with Master: a5b0336c30f648cc9b62d292b6c18368 (127.19.84.1:40439)
I20260131 03:07:59.076313 23444 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.1@KRBTEST.COM'} at 127.19.84.1:33491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:59.123284 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:39853
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:59.205582 23644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:59.205803 23644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:59.205828 23644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:59.207288 23644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:59.207330 23644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:59.207356 23644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:07:59.209118 23644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:39853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:59.209326 23644 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:59.209471 23644 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:59.212823 23650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:59.212843 23652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:59.212863 23649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:59.212898 23644 server_base.cc:1060] running on GCE node
I20260131 03:07:59.213202 23644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:59.213446 23644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:59.214651 23644 hybrid_clock.cc:648] HybridClock initialized: now 1769828879214619 us; error 53 us; skew 500 ppm
Jan 31 03:07:59 dist-test-slave-5nnp krb5kdc[23396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828879, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:59.217388 23644 init.cc:377] Logged in from keytab as kudu/127.19.84.2@KRBTEST.COM (short username kudu)
I20260131 03:07:59.218070 23644 webserver.cc:492] Webserver started at http://127.19.84.2:44247/ using document root <none> and password file <none>
I20260131 03:07:59.218302 23644 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:59.218360 23644 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:59.218499 23644 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:59.219641 23644 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "1c5b3b5cc7ad40c394e52d5deda8554f"
format_stamp: "Formatted at 2026-01-31 03:07:59 on dist-test-slave-5nnp"
server_key: "3e13d59480cf7ee758b587c33df31dde"
server_key_iv: "4d7b19aca896991908b1bd6de646fc57"
server_key_version: "encryptionkey@0"
I20260131 03:07:59.219973 23644 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "1c5b3b5cc7ad40c394e52d5deda8554f"
format_stamp: "Formatted at 2026-01-31 03:07:59 on dist-test-slave-5nnp"
server_key: "3e13d59480cf7ee758b587c33df31dde"
server_key_iv: "4d7b19aca896991908b1bd6de646fc57"
server_key_version: "encryptionkey@0"
I20260131 03:07:59.221309 23644 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260131 03:07:59.222149 23659 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:59.222450 23644 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:07:59.222561 23644 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "1c5b3b5cc7ad40c394e52d5deda8554f"
format_stamp: "Formatted at 2026-01-31 03:07:59 on dist-test-slave-5nnp"
server_key: "3e13d59480cf7ee758b587c33df31dde"
server_key_iv: "4d7b19aca896991908b1bd6de646fc57"
server_key_version: "encryptionkey@0"
I20260131 03:07:59.222651 23644 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:59.231271 23644 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:59.233527 23644 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:59.233682 23644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:59.233867 23644 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:07:59.234326 23644 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:07:59.234388 23644 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:59.234433 23644 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:07:59.234467 23644 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:59.240396 23644 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:44965
I20260131 03:07:59.240434 23772 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:44965 every 8 connection(s)
I20260131 03:07:59.240891 23644 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
Jan 31 03:07:59 dist-test-slave-5nnp krb5kdc[23396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828879, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:59.248271 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 23644
I20260131 03:07:59.248358 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:07:59.248546 19792 external_mini_cluster.cc:1468] Setting key 1439ffbeaae554cd729fade917d937f4
I20260131 03:07:59.250022 23484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:59.241671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:58625 (local address 127.19.84.62:39853)
0131 03:07:59.241741 (+ 70us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:59.241743 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:59.242030 (+ 287us) server_negotiation.cc:408] Connection header received
0131 03:07:59.242483 (+ 453us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:59.242484 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:59.242494 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:59.242558 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:59.243326 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:59.243897 (+ 571us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:59.244449 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:59.244560 (+ 111us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:59.246885 (+ 2325us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:59.246892 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:59.246892 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:59.246900 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:59.248470 (+ 1570us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:59.249018 (+ 548us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:59.249019 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:59.249020 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:59.249042 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:59.249336 (+ 294us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:59.249337 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:59.249337 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:59.249428 (+ 91us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:59.249505 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:59.249795 (+ 290us) server_negotiation.cc:300] Negotiation successful
0131 03:07:59.249925 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260131 03:07:59.250146 23775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:59.241654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:39853 (local address 127.19.84.2:58625)
0131 03:07:59.241914 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:59.241944 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:59.242404 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:59.242607 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:59.242610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:59.242960 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:59.243246 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:59.243249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:59.243990 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:59.243991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:59.244371 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:59.244373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:59.244492 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:59.245361 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:59.245378 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:59.246792 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:59.248557 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:59.248564 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:59.248567 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:59.248863 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:59.249111 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:59.249112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:59.249112 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:59.249207 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:59.249495 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:59.249501 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:59.249654 (+ 153us) client_negotiation.cc:769] Sending connection context
0131 03:07:59.249776 (+ 122us) client_negotiation.cc:240] Negotiation successful
0131 03:07:59.249925 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":95,"threads_started":1}
I20260131 03:07:59.250808 23773 heartbeater.cc:344] Connected to a master server at 127.19.84.62:39853
I20260131 03:07:59.250942 23773 heartbeater.cc:461] Registering TS with master...
I20260131 03:07:59.251180 23773 heartbeater.cc:507] Master 127.19.84.62:39853 requested a full tablet report, sending...
I20260131 03:07:59.251649 23444 ts_manager.cc:194] Registered new tserver with Master: 1c5b3b5cc7ad40c394e52d5deda8554f (127.19.84.2:44965)
I20260131 03:07:59.252202 23444 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.2@KRBTEST.COM'} at 127.19.84.2:58625
WARNING: no policy specified for kudu/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:07:59.298319 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:39853
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:07:59.380906 23780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:07:59.381119 23780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:07:59.381145 23780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:07:59.382618 23780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:07:59.382671 23780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:07:59.382699 23780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:07:59.384395 23780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:39853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:07:59.384593 23780 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:07:59.384773 23780 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:07:59.387617 23788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:59.387632 23786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:07:59.387781 23785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:07:59.387863 23780 server_base.cc:1060] running on GCE node
I20260131 03:07:59.388109 23780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:07:59.388350 23780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:07:59.389513 23780 hybrid_clock.cc:648] HybridClock initialized: now 1769828879389507 us; error 35 us; skew 500 ppm
Jan 31 03:07:59 dist-test-slave-5nnp krb5kdc[23396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828879, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:07:59.392343 23780 init.cc:377] Logged in from keytab as kudu/127.19.84.3@KRBTEST.COM (short username kudu)
I20260131 03:07:59.392946 23780 webserver.cc:492] Webserver started at http://127.19.84.3:45699/ using document root <none> and password file <none>
I20260131 03:07:59.393149 23780 fs_manager.cc:362] Metadata directory not provided
I20260131 03:07:59.393204 23780 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:07:59.393342 23780 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:07:59.394325 23780 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "9eba2a59950e4c4ab6de9effc70f36ea"
format_stamp: "Formatted at 2026-01-31 03:07:59 on dist-test-slave-5nnp"
server_key: "6dc8d273bc6de622e76b781fef613e48"
server_key_iv: "596f11b099d39eb151c73aa22065f500"
server_key_version: "encryptionkey@0"
I20260131 03:07:59.394796 23780 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "9eba2a59950e4c4ab6de9effc70f36ea"
format_stamp: "Formatted at 2026-01-31 03:07:59 on dist-test-slave-5nnp"
server_key: "6dc8d273bc6de622e76b781fef613e48"
server_key_iv: "596f11b099d39eb151c73aa22065f500"
server_key_version: "encryptionkey@0"
I20260131 03:07:59.396180 23780 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260131 03:07:59.396984 23795 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:59.397182 23780 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:07:59.397284 23780 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "9eba2a59950e4c4ab6de9effc70f36ea"
format_stamp: "Formatted at 2026-01-31 03:07:59 on dist-test-slave-5nnp"
server_key: "6dc8d273bc6de622e76b781fef613e48"
server_key_iv: "596f11b099d39eb151c73aa22065f500"
server_key_version: "encryptionkey@0"
I20260131 03:07:59.397368 23780 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:07:59.420969 23780 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:07:59.422899 23780 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:07:59.423069 23780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:07:59.423365 23780 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:07:59.423928 23780 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:07:59.423995 23780 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:59.424034 23780 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:07:59.424082 23780 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:07:59.430337 23780 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:43113
I20260131 03:07:59.430377 23908 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:43113 every 8 connection(s)
I20260131 03:07:59.430712 23780 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
I20260131 03:07:59.433321 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 23780
I20260131 03:07:59.433432 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:07:59.433583 19792 external_mini_cluster.cc:1468] Setting key 47e2f8599647cc08cd415235c54b1462
Jan 31 03:07:59 dist-test-slave-5nnp krb5kdc[23396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828879, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:59.439334 23484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:59.431475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:39935 (local address 127.19.84.62:39853)
0131 03:07:59.431539 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:59.431583 (+ 44us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:59.431752 (+ 169us) server_negotiation.cc:408] Connection header received
0131 03:07:59.432282 (+ 530us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:59.432283 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:59.432296 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:59.432355 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:59.433149 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:59.433751 (+ 602us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:59.434435 (+ 684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:59.434627 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:59.436877 (+ 2250us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:59.436880 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:59.436881 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:59.436887 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:59.438180 (+ 1293us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:59.438596 (+ 416us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:59.438597 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:59.438597 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:59.438615 (+ 18us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:59.438812 (+ 197us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:59.438812 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:59.438813 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:59.438913 (+ 100us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:59.438963 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:59.439198 (+ 235us) server_negotiation.cc:300] Negotiation successful
0131 03:07:59.439266 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260131 03:07:59.439517 23911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:59.431457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:39853 (local address 127.19.84.3:39935)
0131 03:07:59.431643 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0131 03:07:59.431671 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0131 03:07:59.432168 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:07:59.432422 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:07:59.432425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:07:59.432743 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:07:59.433024 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:59.433027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:59.433851 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:07:59.433852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:07:59.434342 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:07:59.434345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:07:59.434487 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:59.435361 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:07:59.435400 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:07:59.436786 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:07:59.438271 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:59.438276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:59.438278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:59.438498 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:59.438671 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:07:59.438671 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:07:59.438672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:07:59.438755 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:07:59.438967 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:07:59.438971 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:07:59.439088 (+ 117us) client_negotiation.cc:769] Sending connection context
0131 03:07:59.439175 (+ 87us) client_negotiation.cc:240] Negotiation successful
0131 03:07:59.439286 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":76,"threads_started":1}
I20260131 03:07:59.440088 23909 heartbeater.cc:344] Connected to a master server at 127.19.84.62:39853
I20260131 03:07:59.440212 23909 heartbeater.cc:461] Registering TS with master...
I20260131 03:07:59.440460 23909 heartbeater.cc:507] Master 127.19.84.62:39853 requested a full tablet report, sending...
I20260131 03:07:59.440900 23444 ts_manager.cc:194] Registered new tserver with Master: 9eba2a59950e4c4ab6de9effc70f36ea (127.19.84.3:43113)
I20260131 03:07:59.441270 23444 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:39935
I20260131 03:07:59.445454 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Jan 31 03:07:59 dist-test-slave-5nnp krb5kdc[23396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828879, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jan 31 03:07:59 dist-test-slave-5nnp krb5kdc[23396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828879, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:07:59.469851 23484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:07:59.462544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40342 (local address 127.19.84.62:39853)
0131 03:07:59.462627 (+ 83us) server_negotiation.cc:207] Beginning negotiation
0131 03:07:59.462629 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:07:59.462710 (+ 81us) server_negotiation.cc:408] Connection header received
0131 03:07:59.462757 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:07:59.462758 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:07:59.462767 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:07:59.462830 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:07:59.463651 (+ 821us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:59.464149 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:07:59.464772 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:07:59.464908 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:07:59.467026 (+ 2118us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:07:59.467032 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:07:59.467032 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:07:59.467041 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:07:59.468700 (+ 1659us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:59.469063 (+ 363us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:59.469064 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:59.469065 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:59.469096 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:07:59.469264 (+ 168us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:07:59.469265 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:07:59.469266 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:07:59.469395 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:07:59.469508 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0131 03:07:59.469643 (+ 135us) server_negotiation.cc:300] Negotiation successful
0131 03:07:59.469729 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260131 03:07:59.470099 23444 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40342
I20260131 03:07:59.470553 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 23508
I20260131 03:07:59.475509 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 23644
I20260131 03:07:59.480021 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 23780
I20260131 03:07:59.484444 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 23412
2026-01-31T03:07:59Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2963 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:07:59 dist-test-slave-5nnp krb5kdc[23921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:07:59 dist-test-slave-5nnp krb5kdc[23921](info): set up 2 sockets
Jan 31 03:07:59 dist-test-slave-5nnp krb5kdc[23921](info): commencing operation
krb5kdc: starting...
W20260131 03:08:01.532331 19792 mini_kdc.cc:121] Time spent starting KDC: real 2.027s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:08:01 dist-test-slave-5nnp krb5kdc[23921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:08:01Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:08:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:08:01.683914 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:39019
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:39019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:01.768044 23937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:01.768227 23937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:01.768250 23937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:01.769685 23937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:08:01.769732 23937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:01.769747 23937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:01.769759 23937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:08:01.769771 23937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:08:01.771438 23937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:39019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:39019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:01.771632 23937 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:01.771770 23937 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:01.774466 23945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:01.774500 23943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:01.774475 23942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:01.774848 23937 server_base.cc:1060] running on GCE node
I20260131 03:08:01.775099 23937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:01.775362 23937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:01.776592 23937 hybrid_clock.cc:648] HybridClock initialized: now 1769828881776577 us; error 44 us; skew 500 ppm
Jan 31 03:08:01 dist-test-slave-5nnp krb5kdc[23921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828881, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:01.779042 23937 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:08:01.779778 23937 webserver.cc:492] Webserver started at http://127.19.84.62:43829/ using document root <none> and password file <none>
I20260131 03:08:01.780014 23937 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:01.780074 23937 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:01.780210 23937 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:01.781221 23937 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "303da58863ec4696938d0ef357543aee"
format_stamp: "Formatted at 2026-01-31 03:08:01 on dist-test-slave-5nnp"
server_key: "1db73b1bf406260b31c98313599fd49b"
server_key_iv: "2b1d1e38ad237977e29b4a76c79a7696"
server_key_version: "encryptionkey@0"
I20260131 03:08:01.781561 23937 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "303da58863ec4696938d0ef357543aee"
format_stamp: "Formatted at 2026-01-31 03:08:01 on dist-test-slave-5nnp"
server_key: "1db73b1bf406260b31c98313599fd49b"
server_key_iv: "2b1d1e38ad237977e29b4a76c79a7696"
server_key_version: "encryptionkey@0"
I20260131 03:08:01.783030 23937 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260131 03:08:01.783869 23952 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:01.784092 23937 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260131 03:08:01.784214 23937 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "303da58863ec4696938d0ef357543aee"
format_stamp: "Formatted at 2026-01-31 03:08:01 on dist-test-slave-5nnp"
server_key: "1db73b1bf406260b31c98313599fd49b"
server_key_iv: "2b1d1e38ad237977e29b4a76c79a7696"
server_key_version: "encryptionkey@0"
I20260131 03:08:01.784303 23937 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:01.830569 23937 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:01.836402 23937 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:01.836593 23937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:01.840663 23937 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:39019
I20260131 03:08:01.840715 24004 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:39019 every 8 connection(s)
I20260131 03:08:01.841081 23937 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:08:01.841806 24005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:01.844416 24005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 303da58863ec4696938d0ef357543aee: Bootstrap starting.
I20260131 03:08:01.844972 24005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 303da58863ec4696938d0ef357543aee: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:01.845178 24005 log.cc:826] T 00000000000000000000000000000000 P 303da58863ec4696938d0ef357543aee: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:01.846024 24005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 303da58863ec4696938d0ef357543aee: No bootstrap required, opened a new log
I20260131 03:08:01.847277 24005 raft_consensus.cc:359] T 00000000000000000000000000000000 P 303da58863ec4696938d0ef357543aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "303da58863ec4696938d0ef357543aee" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 39019 } }
I20260131 03:08:01.847489 24005 raft_consensus.cc:385] T 00000000000000000000000000000000 P 303da58863ec4696938d0ef357543aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:01.847517 24005 raft_consensus.cc:740] T 00000000000000000000000000000000 P 303da58863ec4696938d0ef357543aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 303da58863ec4696938d0ef357543aee, State: Initialized, Role: FOLLOWER
I20260131 03:08:01.847656 24005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 303da58863ec4696938d0ef357543aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "303da58863ec4696938d0ef357543aee" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 39019 } }
I20260131 03:08:01.847764 24005 raft_consensus.cc:399] T 00000000000000000000000000000000 P 303da58863ec4696938d0ef357543aee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:08:01.847824 24005 raft_consensus.cc:493] T 00000000000000000000000000000000 P 303da58863ec4696938d0ef357543aee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:08:01.847885 24005 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 303da58863ec4696938d0ef357543aee [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:01.848397 24005 raft_consensus.cc:515] T 00000000000000000000000000000000 P 303da58863ec4696938d0ef357543aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "303da58863ec4696938d0ef357543aee" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 39019 } }
I20260131 03:08:01.848495 24005 leader_election.cc:304] T 00000000000000000000000000000000 P 303da58863ec4696938d0ef357543aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 303da58863ec4696938d0ef357543aee; no voters:
I20260131 03:08:01.848625 24005 leader_election.cc:290] T 00000000000000000000000000000000 P 303da58863ec4696938d0ef357543aee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:08:01.848697 24008 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 303da58863ec4696938d0ef357543aee [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:08:01.848860 24008 raft_consensus.cc:697] T 00000000000000000000000000000000 P 303da58863ec4696938d0ef357543aee [term 1 LEADER]: Becoming Leader. State: Replica: 303da58863ec4696938d0ef357543aee, State: Running, Role: LEADER
I20260131 03:08:01.848899 24005 sys_catalog.cc:565] T 00000000000000000000000000000000 P 303da58863ec4696938d0ef357543aee [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:08:01.849112 24008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 303da58863ec4696938d0ef357543aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "303da58863ec4696938d0ef357543aee" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 39019 } }
I20260131 03:08:01.849717 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 23937
I20260131 03:08:01.849807 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:08:01.849977 19792 external_mini_cluster.cc:1468] Setting key 379d1131de2c0c211be3a93973b5feb1
I20260131 03:08:01.850296 24009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 303da58863ec4696938d0ef357543aee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "303da58863ec4696938d0ef357543aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "303da58863ec4696938d0ef357543aee" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 39019 } } }
I20260131 03:08:01.850622 24009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 303da58863ec4696938d0ef357543aee [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:01.850550 24010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 303da58863ec4696938d0ef357543aee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 303da58863ec4696938d0ef357543aee. Latest consensus state: current_term: 1 leader_uuid: "303da58863ec4696938d0ef357543aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "303da58863ec4696938d0ef357543aee" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 39019 } } }
I20260131 03:08:01.850908 24010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 303da58863ec4696938d0ef357543aee [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:01.851233 24020 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:08:01.851926 24020 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:08:01.853387 24020 catalog_manager.cc:1357] Generated new cluster ID: b43aa82d5e6b45a29149c592c25ff427
I20260131 03:08:01.853482 24020 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Jan 31 03:08:01 dist-test-slave-5nnp krb5kdc[23921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:08:01.861389 24019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:01.850687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59920 (local address 127.19.84.62:39019)
0131 03:08:01.850864 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:01.850880 (+ 16us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:01.851136 (+ 256us) server_negotiation.cc:408] Connection header received
0131 03:08:01.851284 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:01.851290 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:01.851343 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:01.851520 (+ 177us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:01.852372 (+ 852us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:01.853249 (+ 877us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:01.853868 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:01.854142 (+ 274us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:01.857814 (+ 3672us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:01.857821 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:01.857821 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:01.857848 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:01.859940 (+ 2092us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:01.860320 (+ 380us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:01.860323 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:01.860323 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:01.860384 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:01.860557 (+ 173us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:01.860558 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:01.860559 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:01.860775 (+ 216us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:01.860943 (+ 168us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:01.861044 (+ 101us) server_negotiation.cc:300] Negotiation successful
0131 03:08:01.861189 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260131 03:08:01.866264 24020 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:08:01.867350 24020 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:08:01.877673 24020 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 303da58863ec4696938d0ef357543aee: Generated new TSK 0
I20260131 03:08:01.877925 24020 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:08:01.936167 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:39019
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:02.031376 24033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:02.032084 24033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:02.032193 24033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:02.035322 24033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:02.035424 24033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:02.035480 24033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:08:02.038609 24033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:39019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:02.038962 24033 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:02.039178 24033 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260131 03:08:02.043181 24033 server_base.cc:1060] running on GCE node
W20260131 03:08:02.043257 24038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:02.043516 24039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:02.043561 24041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:02.043949 24033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:02.044246 24033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:02.045475 24033 hybrid_clock.cc:648] HybridClock initialized: now 1769828882045438 us; error 56 us; skew 500 ppm
Jan 31 03:08:02 dist-test-slave-5nnp krb5kdc[23921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828882, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:02.048628 24033 init.cc:377] Logged in from keytab as kudu/127.19.84.1@KRBTEST.COM (short username kudu)
I20260131 03:08:02.049450 24033 webserver.cc:492] Webserver started at http://127.19.84.1:33681/ using document root <none> and password file <none>
I20260131 03:08:02.049678 24033 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:02.049736 24033 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:02.049871 24033 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:02.050796 24033 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "f52c99df0f4d49d2a886983aba896333"
format_stamp: "Formatted at 2026-01-31 03:08:02 on dist-test-slave-5nnp"
server_key: "38f5c55c980d5e7d53f2e71fa8ae8121"
server_key_iv: "a3f2ad313ddc8f1fd4c1b2582d3d9318"
server_key_version: "encryptionkey@0"
I20260131 03:08:02.051085 24033 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "f52c99df0f4d49d2a886983aba896333"
format_stamp: "Formatted at 2026-01-31 03:08:02 on dist-test-slave-5nnp"
server_key: "38f5c55c980d5e7d53f2e71fa8ae8121"
server_key_iv: "a3f2ad313ddc8f1fd4c1b2582d3d9318"
server_key_version: "encryptionkey@0"
I20260131 03:08:02.052526 24033 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:08:02.053347 24048 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:02.053543 24033 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:02.053627 24033 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "f52c99df0f4d49d2a886983aba896333"
format_stamp: "Formatted at 2026-01-31 03:08:02 on dist-test-slave-5nnp"
server_key: "38f5c55c980d5e7d53f2e71fa8ae8121"
server_key_iv: "a3f2ad313ddc8f1fd4c1b2582d3d9318"
server_key_version: "encryptionkey@0"
I20260131 03:08:02.053687 24033 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:02.063354 24033 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:02.065726 24033 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:02.065840 24033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:02.066051 24033 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:02.066411 24033 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:02.066471 24033 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:02.066515 24033 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:02.066547 24033 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:02.072433 24033 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:33053
I20260131 03:08:02.072500 24161 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:33053 every 8 connection(s)
I20260131 03:08:02.072834 24033 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
Jan 31 03:08:02 dist-test-slave-5nnp krb5kdc[23921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828882, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:08:02.081938 24019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:02.073629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:41721 (local address 127.19.84.62:39019)
0131 03:08:02.073712 (+ 83us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:02.073713 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:02.073927 (+ 214us) server_negotiation.cc:408] Connection header received
0131 03:08:02.074366 (+ 439us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:02.074368 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:02.074383 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:02.074456 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:02.075278 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:02.075923 (+ 645us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:02.076566 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:02.076690 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:02.078977 (+ 2287us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:02.078982 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:02.078982 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:02.078991 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:02.080679 (+ 1688us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:02.081074 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:02.081075 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:02.081075 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:02.081099 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:02.081325 (+ 226us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:02.081326 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:02.081326 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:02.081429 (+ 103us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:02.081484 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:02.081739 (+ 255us) server_negotiation.cc:300] Negotiation successful
0131 03:08:02.081855 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260131 03:08:02.082029 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 24033
I20260131 03:08:02.082093 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:02.073574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:39019 (local address 127.19.84.1:41721)
0131 03:08:02.073789 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:02.073814 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:02.074271 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:02.074534 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:02.074537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:02.074881 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:02.075160 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:02.075163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:02.076035 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:02.076036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:02.076437 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:02.076439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:02.076840 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:02.077376 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:02.077396 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:02.078874 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:02.080774 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:02.080780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:02.080782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:02.080997 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:02.081186 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:02.081187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:02.081187 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:02.081258 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:02.081486 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:02.081490 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:02.081609 (+ 119us) client_negotiation.cc:769] Sending connection context
0131 03:08:02.081706 (+ 97us) client_negotiation.cc:240] Negotiation successful
0131 03:08:02.081862 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":71,"threads_started":1}
I20260131 03:08:02.082123 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:08:02.082288 19792 external_mini_cluster.cc:1468] Setting key 12dfef76b227745779d8cd358284ab0b
I20260131 03:08:02.082682 24162 heartbeater.cc:344] Connected to a master server at 127.19.84.62:39019
I20260131 03:08:02.082974 24162 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:02.083227 24162 heartbeater.cc:507] Master 127.19.84.62:39019 requested a full tablet report, sending...
I20260131 03:08:02.083823 23969 ts_manager.cc:194] Registered new tserver with Master: f52c99df0f4d49d2a886983aba896333 (127.19.84.1:33053)
I20260131 03:08:02.084666 23969 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.1@KRBTEST.COM'} at 127.19.84.1:41721
WARNING: no policy specified for kudu/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:08:02.134925 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:39019
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:02.215724 24169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:02.215956 24169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:02.215986 24169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:02.217519 24169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:02.217573 24169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:02.217603 24169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:08:02.219842 24169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:39019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:02.220043 24169 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:02.220201 24169 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:02.223206 24175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:02.223206 24177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:02.223524 24174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:02.224097 24169 server_base.cc:1060] running on GCE node
I20260131 03:08:02.224273 24169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:02.224459 24169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:02.225605 24169 hybrid_clock.cc:648] HybridClock initialized: now 1769828882225583 us; error 40 us; skew 500 ppm
Jan 31 03:08:02 dist-test-slave-5nnp krb5kdc[23921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828882, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:02.228919 24169 init.cc:377] Logged in from keytab as kudu/127.19.84.2@KRBTEST.COM (short username kudu)
I20260131 03:08:02.229666 24169 webserver.cc:492] Webserver started at http://127.19.84.2:37245/ using document root <none> and password file <none>
I20260131 03:08:02.229893 24169 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:02.229954 24169 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:02.230082 24169 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:02.231081 24169 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "be04a62fb47d4e93bb689a64a135ab91"
format_stamp: "Formatted at 2026-01-31 03:08:02 on dist-test-slave-5nnp"
server_key: "cf8d1c733215ef0a828f1ba7d9fef17f"
server_key_iv: "63a7d09317dd90b69bf1819d9b331e6f"
server_key_version: "encryptionkey@0"
I20260131 03:08:02.231477 24169 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "be04a62fb47d4e93bb689a64a135ab91"
format_stamp: "Formatted at 2026-01-31 03:08:02 on dist-test-slave-5nnp"
server_key: "cf8d1c733215ef0a828f1ba7d9fef17f"
server_key_iv: "63a7d09317dd90b69bf1819d9b331e6f"
server_key_version: "encryptionkey@0"
I20260131 03:08:02.232800 24169 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:08:02.233680 24184 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:02.233886 24169 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:08:02.233979 24169 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "be04a62fb47d4e93bb689a64a135ab91"
format_stamp: "Formatted at 2026-01-31 03:08:02 on dist-test-slave-5nnp"
server_key: "cf8d1c733215ef0a828f1ba7d9fef17f"
server_key_iv: "63a7d09317dd90b69bf1819d9b331e6f"
server_key_version: "encryptionkey@0"
I20260131 03:08:02.234064 24169 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:02.243767 24169 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:02.246021 24169 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:02.246183 24169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:02.246421 24169 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:02.246750 24169 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:02.246803 24169 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:02.246848 24169 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:02.246905 24169 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:02.253561 24169 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:35583
I20260131 03:08:02.253618 24297 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:35583 every 8 connection(s)
I20260131 03:08:02.253948 24169 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
Jan 31 03:08:02 dist-test-slave-5nnp krb5kdc[23921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828882, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:08:02.259867 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 24169
I20260131 03:08:02.259969 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:08:02.260174 19792 external_mini_cluster.cc:1468] Setting key e5a73659183fc520a8a5318df3d4db55
I20260131 03:08:02.263175 24019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:02.254717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:49223 (local address 127.19.84.62:39019)
0131 03:08:02.254808 (+ 91us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:02.254810 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:02.254975 (+ 165us) server_negotiation.cc:408] Connection header received
0131 03:08:02.255560 (+ 585us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:02.255561 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:02.255575 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:02.255646 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:02.256414 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:02.256955 (+ 541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:02.257585 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:02.257786 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:02.260113 (+ 2327us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:02.260118 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:02.260118 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:02.260126 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:02.261669 (+ 1543us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:02.262183 (+ 514us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:02.262184 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:02.262184 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:02.262209 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:02.262479 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:02.262480 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:02.262481 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:02.262606 (+ 125us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:02.262686 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:02.262980 (+ 294us) server_negotiation.cc:300] Negotiation successful
0131 03:08:02.263084 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260131 03:08:02.263345 24300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:02.254675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:39019 (local address 127.19.84.2:49223)
0131 03:08:02.254846 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:02.254892 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:02.255459 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:02.255703 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:02.255707 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:02.256039 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:02.256312 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:02.256316 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:02.257052 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:02.257054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:02.257503 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:02.257505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:02.257963 (+ 458us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:02.258447 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:02.258463 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:02.260021 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:02.261757 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:02.261763 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:02.261766 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:02.262090 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:02.262331 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:02.262332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:02.262333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:02.262409 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:02.262677 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:02.262682 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:02.262837 (+ 155us) client_negotiation.cc:769] Sending connection context
0131 03:08:02.262962 (+ 125us) client_negotiation.cc:240] Negotiation successful
0131 03:08:02.263105 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
I20260131 03:08:02.263878 24298 heartbeater.cc:344] Connected to a master server at 127.19.84.62:39019
I20260131 03:08:02.263969 24298 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:02.264200 24298 heartbeater.cc:507] Master 127.19.84.62:39019 requested a full tablet report, sending...
I20260131 03:08:02.264616 23969 ts_manager.cc:194] Registered new tserver with Master: be04a62fb47d4e93bb689a64a135ab91 (127.19.84.2:35583)
I20260131 03:08:02.265012 23969 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.2@KRBTEST.COM'} at 127.19.84.2:49223
WARNING: no policy specified for kudu/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:08:02.311272 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:39019
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:02.395946 24305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:02.396127 24305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:02.396153 24305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:02.397658 24305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:02.397702 24305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:02.397729 24305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:08:02.399348 24305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:39019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:02.399626 24305 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:02.399780 24305 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:02.402657 24313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:02.402694 24310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:02.402698 24311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:02.402998 24305 server_base.cc:1060] running on GCE node
I20260131 03:08:02.403223 24305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:02.403563 24305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:02.404732 24305 hybrid_clock.cc:648] HybridClock initialized: now 1769828882404707 us; error 39 us; skew 500 ppm
Jan 31 03:08:02 dist-test-slave-5nnp krb5kdc[23921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828882, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:02.407601 24305 init.cc:377] Logged in from keytab as kudu/127.19.84.3@KRBTEST.COM (short username kudu)
I20260131 03:08:02.408208 24305 webserver.cc:492] Webserver started at http://127.19.84.3:39693/ using document root <none> and password file <none>
I20260131 03:08:02.408407 24305 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:02.408444 24305 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:02.408567 24305 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:02.409510 24305 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "952e1992cc064227ae7e1ecf2c5d2eaf"
format_stamp: "Formatted at 2026-01-31 03:08:02 on dist-test-slave-5nnp"
server_key: "127474fcce1f88448993ee38766d13f6"
server_key_iv: "5f0f95e5cdcdcfe73caef1307a83a05e"
server_key_version: "encryptionkey@0"
I20260131 03:08:02.409839 24305 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "952e1992cc064227ae7e1ecf2c5d2eaf"
format_stamp: "Formatted at 2026-01-31 03:08:02 on dist-test-slave-5nnp"
server_key: "127474fcce1f88448993ee38766d13f6"
server_key_iv: "5f0f95e5cdcdcfe73caef1307a83a05e"
server_key_version: "encryptionkey@0"
I20260131 03:08:02.411111 24305 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:08:02.412014 24320 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:02.412199 24305 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:02.412288 24305 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "952e1992cc064227ae7e1ecf2c5d2eaf"
format_stamp: "Formatted at 2026-01-31 03:08:02 on dist-test-slave-5nnp"
server_key: "127474fcce1f88448993ee38766d13f6"
server_key_iv: "5f0f95e5cdcdcfe73caef1307a83a05e"
server_key_version: "encryptionkey@0"
I20260131 03:08:02.412371 24305 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:02.427711 24305 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:02.429615 24305 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:02.429778 24305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:02.430090 24305 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:02.430418 24305 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:02.430469 24305 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:02.430524 24305 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:02.430577 24305 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:02.436448 24305 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:46611
I20260131 03:08:02.436484 24433 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:46611 every 8 connection(s)
I20260131 03:08:02.436817 24305 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
Jan 31 03:08:02 dist-test-slave-5nnp krb5kdc[23921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828882, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:08:02.445629 24019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:02.437637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:58717 (local address 127.19.84.62:39019)
0131 03:08:02.437727 (+ 90us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:02.437730 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:02.437882 (+ 152us) server_negotiation.cc:408] Connection header received
0131 03:08:02.438342 (+ 460us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:02.438343 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:02.438352 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:02.438419 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:02.439183 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:02.439753 (+ 570us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:02.440318 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:02.440438 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:02.442768 (+ 2330us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:02.442773 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:02.442773 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:02.442782 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:02.444162 (+ 1380us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:02.444577 (+ 415us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:02.444578 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:02.444579 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:02.444610 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:02.444815 (+ 205us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:02.444816 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:02.444816 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:02.444927 (+ 111us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:02.445028 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:02.445391 (+ 363us) server_negotiation.cc:300] Negotiation successful
0131 03:08:02.445522 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260131 03:08:02.445744 24436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:02.437548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:39019 (local address 127.19.84.3:58717)
0131 03:08:02.437756 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:02.437794 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:02.438265 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:02.438464 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:02.438468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:02.438793 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:02.439079 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:02.439082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:02.439836 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:02.439837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:02.440244 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:02.440246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:02.440605 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:02.441177 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:02.441197 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:02.442653 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:02.444266 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:02.444272 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:02.444275 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:02.444489 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:02.444687 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:02.444688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:02.444688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:02.444755 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:02.445019 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:02.445024 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:02.445179 (+ 155us) client_negotiation.cc:769] Sending connection context
0131 03:08:02.445299 (+ 120us) client_negotiation.cc:240] Negotiation successful
0131 03:08:02.445533 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":126,"threads_started":1}
I20260131 03:08:02.446256 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 24305
I20260131 03:08:02.446348 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:08:02.446432 24434 heartbeater.cc:344] Connected to a master server at 127.19.84.62:39019
I20260131 03:08:02.446534 19792 external_mini_cluster.cc:1468] Setting key 385e5ed6e435a26ea3b9c4125c4739dc
I20260131 03:08:02.446550 24434 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:02.446787 24434 heartbeater.cc:507] Master 127.19.84.62:39019 requested a full tablet report, sending...
I20260131 03:08:02.447178 23969 ts_manager.cc:194] Registered new tserver with Master: 952e1992cc064227ae7e1ecf2c5d2eaf (127.19.84.3:46611)
I20260131 03:08:02.447466 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:08:02.447697 23969 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:58717
Jan 31 03:08:02 dist-test-slave-5nnp krb5kdc[23921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 31 03:08:02 dist-test-slave-5nnp krb5kdc[23921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.1@KRBTEST.COM
I20260131 03:08:02.470486 24441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:02.462422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52062 (local address 127.19.84.1:33053)
0131 03:08:02.462694 (+ 272us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:02.462703 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:02.462717 (+ 14us) server_negotiation.cc:408] Connection header received
0131 03:08:02.462784 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:02.462787 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:02.462804 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:02.462912 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:02.463941 (+ 1029us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:02.464994 (+ 1053us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:02.465559 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:02.465749 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:02.467825 (+ 2076us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:02.467835 (+ 10us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:02.467835 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:02.467843 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:02.469576 (+ 1733us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:02.469906 (+ 330us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:02.469909 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:02.469909 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:02.469941 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:02.470073 (+ 132us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:02.470073 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:02.470073 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:02.470167 (+ 94us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:02.470269 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:02.470332 (+ 63us) server_negotiation.cc:300] Negotiation successful
0131 03:08:02.470401 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":125,"threads_started":1}
Jan 31 03:08:02 dist-test-slave-5nnp krb5kdc[23921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:08:02.478746 24019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:02.471918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59936 (local address 127.19.84.62:39019)
0131 03:08:02.471983 (+ 65us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:02.471985 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:02.471993 (+ 8us) server_negotiation.cc:408] Connection header received
0131 03:08:02.472014 (+ 21us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:02.472015 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:02.472023 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:02.472070 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:02.472672 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:02.473148 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:02.473811 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:02.473949 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:02.476251 (+ 2302us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:02.476256 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:02.476256 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:02.476263 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:02.477698 (+ 1435us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:02.478020 (+ 322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:02.478021 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:02.478022 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:02.478044 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:02.478228 (+ 184us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:02.478229 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:02.478229 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:02.478347 (+ 118us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:02.478429 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:02.478579 (+ 150us) server_negotiation.cc:300] Negotiation successful
0131 03:08:02.478675 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260131 03:08:02.479029 23969 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59936
I20260131 03:08:02.479597 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 24033
I20260131 03:08:02.484568 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 24169
I20260131 03:08:02.489264 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 24305
I20260131 03:08:02.494208 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 23937
2026-01-31T03:08:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3010 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:08:02 dist-test-slave-5nnp krb5kdc[24446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:08:02 dist-test-slave-5nnp krb5kdc[24446](info): set up 2 sockets
Jan 31 03:08:02 dist-test-slave-5nnp krb5kdc[24446](info): commencing operation
krb5kdc: starting...
W20260131 03:08:04.520165 19792 mini_kdc.cc:121] Time spent starting KDC: real 2.004s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:08:04 dist-test-slave-5nnp krb5kdc[24446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:08:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:08:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:08:04.663195 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:32785
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:36553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:32785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:04.741636 24462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:04.741874 24462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:04.741918 24462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:04.743446 24462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:08:04.743528 24462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:04.743551 24462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:04.743570 24462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:08:04.743597 24462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:08:04.745273 24462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:36553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:32785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:32785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:04.745553 24462 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:04.745764 24462 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:04.748366 24467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:04.748507 24462 server_base.cc:1060] running on GCE node
W20260131 03:08:04.748593 24470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:04.748467 24468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:04.748878 24462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:04.749090 24462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:04.750238 24462 hybrid_clock.cc:648] HybridClock initialized: now 1769828884750214 us; error 40 us; skew 500 ppm
Jan 31 03:08:04 dist-test-slave-5nnp krb5kdc[24446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828884, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:04.753006 24462 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:08:04.753347 24462 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:04.753392 24462 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:04.753523 24462 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:04.754498 24462 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "ec509b3c3a9540a184210589c0633103"
format_stamp: "Formatted at 2026-01-31 03:08:04 on dist-test-slave-5nnp"
server_key: "5f971ea9c25876aabe4e75a5ffa34af3"
server_key_iv: "eb7a37ae7c7c2be12afb8665211bb689"
server_key_version: "encryptionkey@0"
I20260131 03:08:04.754841 24462 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "ec509b3c3a9540a184210589c0633103"
format_stamp: "Formatted at 2026-01-31 03:08:04 on dist-test-slave-5nnp"
server_key: "5f971ea9c25876aabe4e75a5ffa34af3"
server_key_iv: "eb7a37ae7c7c2be12afb8665211bb689"
server_key_version: "encryptionkey@0"
I20260131 03:08:04.756215 24462 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:08:04.757078 24476 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:04.757329 24462 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:04.757428 24462 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "ec509b3c3a9540a184210589c0633103"
format_stamp: "Formatted at 2026-01-31 03:08:04 on dist-test-slave-5nnp"
server_key: "5f971ea9c25876aabe4e75a5ffa34af3"
server_key_iv: "eb7a37ae7c7c2be12afb8665211bb689"
server_key_version: "encryptionkey@0"
I20260131 03:08:04.757511 24462 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:04.772125 24462 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:04.774077 24462 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:04.774227 24462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:04.778304 24462 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:32785
I20260131 03:08:04.778367 24528 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:32785 every 8 connection(s)
I20260131 03:08:04.778656 24462 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:08:04.779312 24529 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:04.781755 24529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec509b3c3a9540a184210589c0633103: Bootstrap starting.
I20260131 03:08:04.782368 24529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec509b3c3a9540a184210589c0633103: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:04.782646 24529 log.cc:826] T 00000000000000000000000000000000 P ec509b3c3a9540a184210589c0633103: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:04.783334 24529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec509b3c3a9540a184210589c0633103: No bootstrap required, opened a new log
I20260131 03:08:04.784597 24529 raft_consensus.cc:359] T 00000000000000000000000000000000 P ec509b3c3a9540a184210589c0633103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec509b3c3a9540a184210589c0633103" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 32785 } }
I20260131 03:08:04.784715 24529 raft_consensus.cc:385] T 00000000000000000000000000000000 P ec509b3c3a9540a184210589c0633103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:04.784737 24529 raft_consensus.cc:740] T 00000000000000000000000000000000 P ec509b3c3a9540a184210589c0633103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec509b3c3a9540a184210589c0633103, State: Initialized, Role: FOLLOWER
I20260131 03:08:04.784871 24529 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec509b3c3a9540a184210589c0633103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec509b3c3a9540a184210589c0633103" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 32785 } }
I20260131 03:08:04.784945 24529 raft_consensus.cc:399] T 00000000000000000000000000000000 P ec509b3c3a9540a184210589c0633103 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:08:04.784988 24529 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec509b3c3a9540a184210589c0633103 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:08:04.785039 24529 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ec509b3c3a9540a184210589c0633103 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:04.785569 24529 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec509b3c3a9540a184210589c0633103 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec509b3c3a9540a184210589c0633103" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 32785 } }
I20260131 03:08:04.785688 24529 leader_election.cc:304] T 00000000000000000000000000000000 P ec509b3c3a9540a184210589c0633103 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec509b3c3a9540a184210589c0633103; no voters:
I20260131 03:08:04.785869 24529 leader_election.cc:290] T 00000000000000000000000000000000 P ec509b3c3a9540a184210589c0633103 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:08:04.785962 24532 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ec509b3c3a9540a184210589c0633103 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:08:04.786105 24532 raft_consensus.cc:697] T 00000000000000000000000000000000 P ec509b3c3a9540a184210589c0633103 [term 1 LEADER]: Becoming Leader. State: Replica: ec509b3c3a9540a184210589c0633103, State: Running, Role: LEADER
I20260131 03:08:04.786109 24529 sys_catalog.cc:565] T 00000000000000000000000000000000 P ec509b3c3a9540a184210589c0633103 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:08:04.786224 24532 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec509b3c3a9540a184210589c0633103 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec509b3c3a9540a184210589c0633103" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 32785 } }
I20260131 03:08:04.786562 24534 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec509b3c3a9540a184210589c0633103 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec509b3c3a9540a184210589c0633103. Latest consensus state: current_term: 1 leader_uuid: "ec509b3c3a9540a184210589c0633103" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec509b3c3a9540a184210589c0633103" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 32785 } } }
I20260131 03:08:04.786646 24534 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec509b3c3a9540a184210589c0633103 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:04.786547 24533 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec509b3c3a9540a184210589c0633103 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec509b3c3a9540a184210589c0633103" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec509b3c3a9540a184210589c0633103" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 32785 } } }
I20260131 03:08:04.786706 24533 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec509b3c3a9540a184210589c0633103 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:04.786825 24537 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:08:04.787267 24537 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:08:04.787899 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 24462
I20260131 03:08:04.788012 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:08:04.788182 19792 external_mini_cluster.cc:1468] Setting key 75bd3483e8725c8094645f8fd58960d9
I20260131 03:08:04.789053 24537 catalog_manager.cc:1357] Generated new cluster ID: ef756b8b941d4b9ebd1494cb427ae95d
I20260131 03:08:04.789125 24537 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Jan 31 03:08:04 dist-test-slave-5nnp krb5kdc[24446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:08:04.798560 24552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:04.788688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.19.84.62:32785)
0131 03:08:04.788850 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:04.788861 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:04.788889 (+ 28us) server_negotiation.cc:408] Connection header received
0131 03:08:04.789022 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:04.789028 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:04.789075 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:04.789203 (+ 128us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:04.790285 (+ 1082us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:04.791457 (+ 1172us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:04.792241 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:04.792465 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:04.795018 (+ 2553us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:04.795025 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:04.795025 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:04.795050 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:04.797098 (+ 2048us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:04.797470 (+ 372us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:04.797473 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:04.797473 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:04.797538 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:04.797712 (+ 174us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:04.797714 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:04.797714 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:04.797992 (+ 278us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:04.798162 (+ 170us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:04.798238 (+ 76us) server_negotiation.cc:300] Negotiation successful
0131 03:08:04.798365 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260131 03:08:04.810221 24537 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:08:04.810880 24537 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:08:04.814368 24537 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ec509b3c3a9540a184210589c0633103: Generated new TSK 0
I20260131 03:08:04.814615 24537 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:08:04.862442 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:32785
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:36553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:04.945204 24557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:04.945400 24557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:04.945427 24557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:04.947012 24557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:04.947063 24557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:04.947093 24557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:08:04.948832 24557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:36553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:32785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:04.949054 24557 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:04.949203 24557 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:04.952463 24565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:04.952952 24562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:04.952783 24557 server_base.cc:1060] running on GCE node
W20260131 03:08:04.952459 24563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:04.953299 24557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:04.953574 24557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:04.954744 24557 hybrid_clock.cc:648] HybridClock initialized: now 1769828884954730 us; error 35 us; skew 500 ppm
Jan 31 03:08:04 dist-test-slave-5nnp krb5kdc[24446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828884, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:04.957432 24557 init.cc:377] Logged in from keytab as kudu/127.19.84.1@KRBTEST.COM (short username kudu)
I20260131 03:08:04.957780 24557 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:04.957823 24557 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:04.957923 24557 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:04.958847 24557 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "6c7430f8843742ddbf9e0bdfee5a779f"
format_stamp: "Formatted at 2026-01-31 03:08:04 on dist-test-slave-5nnp"
server_key: "1643fba17b884603823217c8a2b3b951"
server_key_iv: "deafb9a69ab76e19772f882e548d84d6"
server_key_version: "encryptionkey@0"
I20260131 03:08:04.959116 24557 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "6c7430f8843742ddbf9e0bdfee5a779f"
format_stamp: "Formatted at 2026-01-31 03:08:04 on dist-test-slave-5nnp"
server_key: "1643fba17b884603823217c8a2b3b951"
server_key_iv: "deafb9a69ab76e19772f882e548d84d6"
server_key_version: "encryptionkey@0"
I20260131 03:08:04.960402 24557 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260131 03:08:04.961144 24571 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:04.961361 24557 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:08:04.961423 24557 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "6c7430f8843742ddbf9e0bdfee5a779f"
format_stamp: "Formatted at 2026-01-31 03:08:04 on dist-test-slave-5nnp"
server_key: "1643fba17b884603823217c8a2b3b951"
server_key_iv: "deafb9a69ab76e19772f882e548d84d6"
server_key_version: "encryptionkey@0"
I20260131 03:08:04.961468 24557 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:04.975675 24557 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:04.978013 24557 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:04.978129 24557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:04.978322 24557 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:04.978670 24557 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:04.978706 24557 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:04.978754 24557 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:04.978782 24557 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:04.984917 24557 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:38781
I20260131 03:08:04.984975 24684 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:38781 every 8 connection(s)
I20260131 03:08:04.985268 24557 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
I20260131 03:08:04.987208 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 24557
I20260131 03:08:04.987344 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:08:04.987546 19792 external_mini_cluster.cc:1468] Setting key 3c69d18b51a26c29a8183de28899937b
Jan 31 03:08:04 dist-test-slave-5nnp krb5kdc[24446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828884, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:08:04.995451 24552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:04.986290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:47639 (local address 127.19.84.62:32785)
0131 03:08:04.986374 (+ 84us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:04.986376 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:04.986484 (+ 108us) server_negotiation.cc:408] Connection header received
0131 03:08:04.986980 (+ 496us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:04.986981 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:04.987000 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:04.987135 (+ 135us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:04.988190 (+ 1055us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:04.988771 (+ 581us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:04.989941 (+ 1170us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:04.990078 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:04.992402 (+ 2324us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:04.992408 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:04.992408 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:04.992418 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:04.994034 (+ 1616us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:04.994477 (+ 443us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:04.994479 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:04.994479 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:04.994514 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:04.994728 (+ 214us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:04.994729 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:04.994729 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:04.994845 (+ 116us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:04.994918 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:04.995215 (+ 297us) server_negotiation.cc:300] Negotiation successful
0131 03:08:04.995332 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260131 03:08:04.995594 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:04.986151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:32785 (local address 127.19.84.1:47639)
0131 03:08:04.986346 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:04.986376 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:04.986865 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:04.987220 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:04.987226 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:04.987718 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:04.988093 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:04.988098 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:04.988872 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:04.988875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:04.989814 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:04.989817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:04.989986 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:04.990536 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:04.990567 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:04.992297 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:04.994160 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:04.994167 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:04.994171 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:04.994396 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:04.994588 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:04.994589 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:04.994589 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:04.994662 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:04.994919 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:04.994926 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:04.995063 (+ 137us) client_negotiation.cc:769] Sending connection context
0131 03:08:04.995197 (+ 134us) client_negotiation.cc:240] Negotiation successful
0131 03:08:04.995393 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.19.84.2@KRBTEST.COM; defaulting to no policy
I20260131 03:08:04.996330 24685 heartbeater.cc:344] Connected to a master server at 127.19.84.62:32785
I20260131 03:08:04.996466 24685 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:04.996704 24685 heartbeater.cc:507] Master 127.19.84.62:32785 requested a full tablet report, sending...
I20260131 03:08:04.997293 24493 ts_manager.cc:194] Registered new tserver with Master: 6c7430f8843742ddbf9e0bdfee5a779f (127.19.84.1:38781)
I20260131 03:08:04.997975 24493 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.1@KRBTEST.COM'} at 127.19.84.1:47639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:08:05.046737 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:32785
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:36553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:05.127945 24692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:05.128132 24692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:05.128157 24692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:05.129644 24692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:05.129696 24692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:05.129724 24692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:08:05.131430 24692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:36553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:32785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:05.131685 24692 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:05.131915 24692 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:05.135525 24700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:05.135514 24698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:05.135519 24697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:05.135942 24692 server_base.cc:1060] running on GCE node
I20260131 03:08:05.136116 24692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:05.136293 24692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:05.137442 24692 hybrid_clock.cc:648] HybridClock initialized: now 1769828885137427 us; error 28 us; skew 500 ppm
Jan 31 03:08:05 dist-test-slave-5nnp krb5kdc[24446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828885, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:05.140904 24692 init.cc:377] Logged in from keytab as kudu/127.19.84.2@KRBTEST.COM (short username kudu)
I20260131 03:08:05.141253 24692 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:05.141333 24692 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:05.141474 24692 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:05.142454 24692 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "7499984ffd8641f8bc4d0581fce45177"
format_stamp: "Formatted at 2026-01-31 03:08:05 on dist-test-slave-5nnp"
server_key: "e106cfeba17526a4deb0e82da4748ce2"
server_key_iv: "e2b8cf555340ee1cd10c6314ac1df7b0"
server_key_version: "encryptionkey@0"
I20260131 03:08:05.142796 24692 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "7499984ffd8641f8bc4d0581fce45177"
format_stamp: "Formatted at 2026-01-31 03:08:05 on dist-test-slave-5nnp"
server_key: "e106cfeba17526a4deb0e82da4748ce2"
server_key_iv: "e2b8cf555340ee1cd10c6314ac1df7b0"
server_key_version: "encryptionkey@0"
I20260131 03:08:05.144150 24692 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:08:05.144966 24706 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:05.145179 24692 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:05.145278 24692 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "7499984ffd8641f8bc4d0581fce45177"
format_stamp: "Formatted at 2026-01-31 03:08:05 on dist-test-slave-5nnp"
server_key: "e106cfeba17526a4deb0e82da4748ce2"
server_key_iv: "e2b8cf555340ee1cd10c6314ac1df7b0"
server_key_version: "encryptionkey@0"
I20260131 03:08:05.145362 24692 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:05.162313 24692 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:05.164347 24692 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:05.164508 24692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:05.164791 24692 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:05.165299 24692 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:05.165371 24692 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:05.165417 24692 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:05.165447 24692 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:05.171787 24692 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:38183
I20260131 03:08:05.171875 24819 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:38183 every 8 connection(s)
I20260131 03:08:05.172128 24692 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
Jan 31 03:08:05 dist-test-slave-5nnp krb5kdc[24446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828885, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:08:05.181687 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 24692
I20260131 03:08:05.181800 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:08:05.181985 19792 external_mini_cluster.cc:1468] Setting key cb2ce5c18b5f0c8ef49ac2078e5ea6c8
I20260131 03:08:05.182317 24552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:05.172912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:52099 (local address 127.19.84.62:32785)
0131 03:08:05.173003 (+ 91us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:05.173005 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:05.173110 (+ 105us) server_negotiation.cc:408] Connection header received
0131 03:08:05.173578 (+ 468us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:05.173580 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:05.173594 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:05.173656 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:05.174424 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.174946 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:05.176009 (+ 1063us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.176211 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:05.178717 (+ 2506us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:05.178725 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:05.178726 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:05.178736 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:05.180748 (+ 2012us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:05.181186 (+ 438us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:05.181187 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:05.181187 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:05.181211 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:05.181415 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:05.181417 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:05.181417 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:05.181532 (+ 115us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:05.181620 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:05.182087 (+ 467us) server_negotiation.cc:300] Negotiation successful
0131 03:08:05.182189 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260131 03:08:05.182644 24822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:05.172850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:32785 (local address 127.19.84.2:52099)
0131 03:08:05.173000 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:05.173028 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:05.173474 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:05.173718 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:05.173722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:05.174032 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:05.174304 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:05.174306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.175067 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:05.175068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:05.175847 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:05.175850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.175986 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:05.176845 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:05.176868 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:05.178503 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:05.180861 (+ 2358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:05.180869 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:05.180873 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:05.181109 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:05.181274 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:05.181274 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:05.181275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:05.181340 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:05.181634 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:05.181642 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:05.181842 (+ 200us) client_negotiation.cc:769] Sending connection context
0131 03:08:05.181998 (+ 156us) client_negotiation.cc:240] Negotiation successful
0131 03:08:05.182167 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260131 03:08:05.183256 24820 heartbeater.cc:344] Connected to a master server at 127.19.84.62:32785
I20260131 03:08:05.183357 24820 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:05.183722 24820 heartbeater.cc:507] Master 127.19.84.62:32785 requested a full tablet report, sending...
I20260131 03:08:05.184180 24493 ts_manager.cc:194] Registered new tserver with Master: 7499984ffd8641f8bc4d0581fce45177 (127.19.84.2:38183)
I20260131 03:08:05.184585 24493 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.2@KRBTEST.COM'} at 127.19.84.2:52099
WARNING: no policy specified for kudu/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:08:05.236886 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:32785
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:36553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:05.319062 24827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:05.319240 24827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:05.319264 24827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:05.320811 24827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:05.320861 24827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:05.320888 24827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:08:05.322521 24827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:36553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:32785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:05.322772 24827 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:05.322978 24827 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:05.325832 24832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:05.325896 24833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:05.325846 24835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:05.326033 24827 server_base.cc:1060] running on GCE node
I20260131 03:08:05.326295 24827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:05.326526 24827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:05.327699 24827 hybrid_clock.cc:648] HybridClock initialized: now 1769828885327631 us; error 86 us; skew 500 ppm
Jan 31 03:08:05 dist-test-slave-5nnp krb5kdc[24446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828885, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:05.330331 24827 init.cc:377] Logged in from keytab as kudu/127.19.84.3@KRBTEST.COM (short username kudu)
I20260131 03:08:05.330694 24827 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:05.330766 24827 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:05.330912 24827 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:05.332000 24827 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "d07875ff7c444ea5a4335f58f587cb07"
format_stamp: "Formatted at 2026-01-31 03:08:05 on dist-test-slave-5nnp"
server_key: "94bef7169e1531444bc1db6cf66c4f1e"
server_key_iv: "8ab76f211630d7a5cc82dc91f5adc37b"
server_key_version: "encryptionkey@0"
I20260131 03:08:05.332345 24827 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "d07875ff7c444ea5a4335f58f587cb07"
format_stamp: "Formatted at 2026-01-31 03:08:05 on dist-test-slave-5nnp"
server_key: "94bef7169e1531444bc1db6cf66c4f1e"
server_key_iv: "8ab76f211630d7a5cc82dc91f5adc37b"
server_key_version: "encryptionkey@0"
I20260131 03:08:05.333619 24827 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:08:05.334470 24841 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:05.334733 24827 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:05.334825 24827 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "d07875ff7c444ea5a4335f58f587cb07"
format_stamp: "Formatted at 2026-01-31 03:08:05 on dist-test-slave-5nnp"
server_key: "94bef7169e1531444bc1db6cf66c4f1e"
server_key_iv: "8ab76f211630d7a5cc82dc91f5adc37b"
server_key_version: "encryptionkey@0"
I20260131 03:08:05.334914 24827 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:05.352607 24827 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:05.354406 24827 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:05.354561 24827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:05.354812 24827 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:05.355144 24827 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:05.355199 24827 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:05.355239 24827 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:05.355268 24827 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:05.361745 24827 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:40373
I20260131 03:08:05.361824 24955 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:40373 every 8 connection(s)
I20260131 03:08:05.362080 24827 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
Jan 31 03:08:05 dist-test-slave-5nnp krb5kdc[24446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828885, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:08:05.370533 24552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:05.362826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:43081 (local address 127.19.84.62:32785)
0131 03:08:05.362907 (+ 81us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:05.362910 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:05.363066 (+ 156us) server_negotiation.cc:408] Connection header received
0131 03:08:05.363643 (+ 577us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:05.363644 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:05.363657 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:05.363716 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:05.364500 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.365008 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:05.365908 (+ 900us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.366080 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:05.368033 (+ 1953us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:05.368038 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:05.368038 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:05.368046 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:05.369249 (+ 1203us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:05.369642 (+ 393us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:05.369643 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:05.369644 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:05.369676 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:05.369882 (+ 206us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:05.369883 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:05.369883 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:05.369985 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:05.370061 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:05.370346 (+ 285us) server_negotiation.cc:300] Negotiation successful
0131 03:08:05.370439 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260131 03:08:05.370663 24958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:05.362734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:32785 (local address 127.19.84.3:43081)
0131 03:08:05.362938 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:05.362972 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:05.363537 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:05.363797 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:05.363800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:05.364111 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:05.364383 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:05.364386 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.365100 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:05.365101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:05.365799 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:05.365801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.365955 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:05.366493 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:05.366511 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:05.367938 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:05.369346 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:05.369352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:05.369355 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:05.369565 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:05.369746 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:05.369747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:05.369747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:05.369817 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:05.370051 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:05.370056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:05.370188 (+ 132us) client_negotiation.cc:769] Sending connection context
0131 03:08:05.370325 (+ 137us) client_negotiation.cc:240] Negotiation successful
0131 03:08:05.370465 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":105,"threads_started":1}
I20260131 03:08:05.371251 24956 heartbeater.cc:344] Connected to a master server at 127.19.84.62:32785
I20260131 03:08:05.371345 24956 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:05.371568 24956 heartbeater.cc:507] Master 127.19.84.62:32785 requested a full tablet report, sending...
I20260131 03:08:05.371874 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 24827
I20260131 03:08:05.372011 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:08:05.372102 24493 ts_manager.cc:194] Registered new tserver with Master: d07875ff7c444ea5a4335f58f587cb07 (127.19.84.3:40373)
I20260131 03:08:05.372241 19792 external_mini_cluster.cc:1468] Setting key be94dd3cb43f1b6e61ebf146dc466534
I20260131 03:08:05.372606 24493 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:43081
I20260131 03:08:05.373214 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:08:05.380434 24552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:05.374467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46184 (local address 127.19.84.62:32785)
0131 03:08:05.374541 (+ 74us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:05.374544 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:05.374656 (+ 112us) server_negotiation.cc:408] Connection header received
0131 03:08:05.374745 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:05.374746 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:05.374757 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:05.374831 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:05.375679 (+ 848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.376164 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:05.376752 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.376951 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:05.377829 (+ 878us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:05.377833 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:05.377834 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:05.377841 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:05.379295 (+ 1454us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:05.379692 (+ 397us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:05.379693 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:05.379694 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:05.379716 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:05.379896 (+ 180us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:05.379897 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:05.379898 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:05.380015 (+ 117us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:05.380121 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:05.380246 (+ 125us) server_negotiation.cc:300] Negotiation successful
0131 03:08:05.380328 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260131 03:08:05.381903 24493 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:08:05.383148 24493 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:08:05.388609 24967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:05.385421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:38183 (local address 127.0.0.1:33344)
0131 03:08:05.385883 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:05.385894 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:05.385997 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:05.386217 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:05.386220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:05.386223 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:05.386647 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:05.386648 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.387659 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:05.387660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:05.388315 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:05.388317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.388406 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:05.388443 (+ 37us) client_negotiation.cc:769] Sending connection context
0131 03:08:05.388492 (+ 49us) client_negotiation.cc:240] Negotiation successful
0131 03:08:05.388535 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":3,"thread_start_us":72,"threads_started":1}
I20260131 03:08:05.388926 24970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:05.385631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:40373 (local address 127.0.0.1:51688)
0131 03:08:05.385992 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:05.386000 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:05.386045 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:05.386459 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:05.386459 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:05.386461 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:05.386709 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:05.386712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.387864 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:05.387865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:05.388730 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:05.388731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.388788 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:05.388791 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:08:05.388821 (+ 30us) client_negotiation.cc:240] Negotiation successful
0131 03:08:05.388856 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":144,"threads_started":1}
I20260131 03:08:05.389154 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:05.385528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33344 (local address 127.19.84.2:38183)
0131 03:08:05.385944 (+ 416us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:05.385951 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:05.386062 (+ 111us) server_negotiation.cc:408] Connection header received
0131 03:08:05.386104 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:05.386106 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:05.386121 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:05.386186 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:05.386698 (+ 512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.387575 (+ 877us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:05.388444 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.388987 (+ 543us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:05.389044 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:05.389069 (+ 25us) server_negotiation.cc:300] Negotiation successful
0131 03:08:05.389101 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":274,"threads_started":1}
I20260131 03:08:05.389803 24754 tablet_service.cc:1505] Processing CreateTablet for tablet fc9a2f0cc7aa4ed6b81e36e628e5ad7d (DEFAULT_TABLE table=test-table [id=2e9491b1b17c437ca5cdb51b07bfa13f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:08:05.389902 24972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:05.385794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51688 (local address 127.19.84.3:40373)
0131 03:08:05.386314 (+ 520us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:05.386326 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:05.386336 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:08:05.386374 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:05.386376 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:05.386391 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:05.386458 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:05.386831 (+ 373us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.387782 (+ 951us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:05.389157 (+ 1375us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.389704 (+ 547us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:05.389750 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:05.389781 (+ 31us) server_negotiation.cc:300] Negotiation successful
0131 03:08:05.389832 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":76,"threads_started":1}
I20260131 03:08:05.390129 24754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc9a2f0cc7aa4ed6b81e36e628e5ad7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:05.390479 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:05.385655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51614 (local address 127.19.84.1:38781)
0131 03:08:05.385864 (+ 209us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:05.385873 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:05.385976 (+ 103us) server_negotiation.cc:408] Connection header received
0131 03:08:05.386060 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:05.386062 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:05.386081 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:05.386158 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:05.386574 (+ 416us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.388576 (+ 2002us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:05.389823 (+ 1247us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.390333 (+ 510us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:05.390375 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:05.390399 (+ 24us) server_negotiation.cc:300] Negotiation successful
0131 03:08:05.390427 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":130,"threads_started":1}
I20260131 03:08:05.389729 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:05.385421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:38781 (local address 127.0.0.1:51614)
0131 03:08:05.385840 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:05.385892 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:05.385980 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:05.386218 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:05.386220 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:05.386224 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:05.386511 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:05.386513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.388667 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:05.388668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:05.389525 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:05.389527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.389599 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:05.389603 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:08:05.389635 (+ 32us) client_negotiation.cc:240] Negotiation successful
0131 03:08:05.389670 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":68,"threads_started":1}
I20260131 03:08:05.390659 24890 tablet_service.cc:1505] Processing CreateTablet for tablet fc9a2f0cc7aa4ed6b81e36e628e5ad7d (DEFAULT_TABLE table=test-table [id=2e9491b1b17c437ca5cdb51b07bfa13f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:08:05.390937 24890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc9a2f0cc7aa4ed6b81e36e628e5ad7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:05.391086 24619 tablet_service.cc:1505] Processing CreateTablet for tablet fc9a2f0cc7aa4ed6b81e36e628e5ad7d (DEFAULT_TABLE table=test-table [id=2e9491b1b17c437ca5cdb51b07bfa13f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:08:05.391285 24619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc9a2f0cc7aa4ed6b81e36e628e5ad7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:05.393195 24973 tablet_bootstrap.cc:492] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 7499984ffd8641f8bc4d0581fce45177: Bootstrap starting.
I20260131 03:08:05.393464 24974 tablet_bootstrap.cc:492] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 6c7430f8843742ddbf9e0bdfee5a779f: Bootstrap starting.
I20260131 03:08:05.393985 24973 tablet_bootstrap.cc:654] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 7499984ffd8641f8bc4d0581fce45177: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:05.394109 24974 tablet_bootstrap.cc:654] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 6c7430f8843742ddbf9e0bdfee5a779f: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:05.394284 24973 log.cc:826] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 7499984ffd8641f8bc4d0581fce45177: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:05.394352 24975 tablet_bootstrap.cc:492] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07: Bootstrap starting.
I20260131 03:08:05.394407 24974 log.cc:826] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 6c7430f8843742ddbf9e0bdfee5a779f: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:05.394907 24975 tablet_bootstrap.cc:654] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:05.395188 24975 log.cc:826] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:05.395216 24973 tablet_bootstrap.cc:492] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 7499984ffd8641f8bc4d0581fce45177: No bootstrap required, opened a new log
I20260131 03:08:05.395268 24973 ts_tablet_manager.cc:1403] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 7499984ffd8641f8bc4d0581fce45177: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:08:05.395648 24974 tablet_bootstrap.cc:492] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 6c7430f8843742ddbf9e0bdfee5a779f: No bootstrap required, opened a new log
I20260131 03:08:05.395731 24974 ts_tablet_manager.cc:1403] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 6c7430f8843742ddbf9e0bdfee5a779f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:08:05.396708 24973 raft_consensus.cc:359] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 7499984ffd8641f8bc4d0581fce45177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7430f8843742ddbf9e0bdfee5a779f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38781 } } peers { permanent_uuid: "7499984ffd8641f8bc4d0581fce45177" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38183 } } peers { permanent_uuid: "d07875ff7c444ea5a4335f58f587cb07" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 40373 } }
I20260131 03:08:05.396885 24973 raft_consensus.cc:385] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 7499984ffd8641f8bc4d0581fce45177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:05.396922 24973 raft_consensus.cc:740] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 7499984ffd8641f8bc4d0581fce45177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7499984ffd8641f8bc4d0581fce45177, State: Initialized, Role: FOLLOWER
I20260131 03:08:05.397015 24973 consensus_queue.cc:260] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 7499984ffd8641f8bc4d0581fce45177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7430f8843742ddbf9e0bdfee5a779f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38781 } } peers { permanent_uuid: "7499984ffd8641f8bc4d0581fce45177" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38183 } } peers { permanent_uuid: "d07875ff7c444ea5a4335f58f587cb07" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 40373 } }
I20260131 03:08:05.397162 24974 raft_consensus.cc:359] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 6c7430f8843742ddbf9e0bdfee5a779f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7430f8843742ddbf9e0bdfee5a779f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38781 } } peers { permanent_uuid: "7499984ffd8641f8bc4d0581fce45177" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38183 } } peers { permanent_uuid: "d07875ff7c444ea5a4335f58f587cb07" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 40373 } }
I20260131 03:08:05.397282 24974 raft_consensus.cc:385] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 6c7430f8843742ddbf9e0bdfee5a779f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:05.397287 24973 ts_tablet_manager.cc:1434] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 7499984ffd8641f8bc4d0581fce45177: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260131 03:08:05.397310 24974 raft_consensus.cc:740] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 6c7430f8843742ddbf9e0bdfee5a779f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c7430f8843742ddbf9e0bdfee5a779f, State: Initialized, Role: FOLLOWER
I20260131 03:08:05.397317 24820 heartbeater.cc:499] Master 127.19.84.62:32785 was elected leader, sending a full tablet report...
I20260131 03:08:05.397396 24974 consensus_queue.cc:260] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 6c7430f8843742ddbf9e0bdfee5a779f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7430f8843742ddbf9e0bdfee5a779f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38781 } } peers { permanent_uuid: "7499984ffd8641f8bc4d0581fce45177" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38183 } } peers { permanent_uuid: "d07875ff7c444ea5a4335f58f587cb07" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 40373 } }
I20260131 03:08:05.397679 24974 ts_tablet_manager.cc:1434] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 6c7430f8843742ddbf9e0bdfee5a779f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:08:05.397701 24685 heartbeater.cc:499] Master 127.19.84.62:32785 was elected leader, sending a full tablet report...
I20260131 03:08:05.398526 24975 tablet_bootstrap.cc:492] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07: No bootstrap required, opened a new log
I20260131 03:08:05.398600 24975 ts_tablet_manager.cc:1403] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260131 03:08:05.399802 24975 raft_consensus.cc:359] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7430f8843742ddbf9e0bdfee5a779f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38781 } } peers { permanent_uuid: "7499984ffd8641f8bc4d0581fce45177" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38183 } } peers { permanent_uuid: "d07875ff7c444ea5a4335f58f587cb07" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 40373 } }
I20260131 03:08:05.399919 24975 raft_consensus.cc:385] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:05.399940 24975 raft_consensus.cc:740] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d07875ff7c444ea5a4335f58f587cb07, State: Initialized, Role: FOLLOWER
I20260131 03:08:05.400030 24975 consensus_queue.cc:260] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7430f8843742ddbf9e0bdfee5a779f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38781 } } peers { permanent_uuid: "7499984ffd8641f8bc4d0581fce45177" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38183 } } peers { permanent_uuid: "d07875ff7c444ea5a4335f58f587cb07" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 40373 } }
I20260131 03:08:05.400298 24956 heartbeater.cc:499] Master 127.19.84.62:32785 was elected leader, sending a full tablet report...
I20260131 03:08:05.400478 24975 ts_tablet_manager.cc:1434] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260131 03:08:05.422673 24821 tablet.cc:2378] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 7499984ffd8641f8bc4d0581fce45177: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260131 03:08:05.486208 24686 tablet.cc:2378] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 6c7430f8843742ddbf9e0bdfee5a779f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:08:05.584633 24981 raft_consensus.cc:493] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:08:05.584755 24981 raft_consensus.cc:515] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7430f8843742ddbf9e0bdfee5a779f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38781 } } peers { permanent_uuid: "7499984ffd8641f8bc4d0581fce45177" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38183 } } peers { permanent_uuid: "d07875ff7c444ea5a4335f58f587cb07" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 40373 } }
I20260131 03:08:05.585211 24981 leader_election.cc:290] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c7430f8843742ddbf9e0bdfee5a779f (127.19.84.1:38781), 7499984ffd8641f8bc4d0581fce45177 (127.19.84.2:38183)
I20260131 03:08:05.587692 24958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:05.585394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:38183 (local address 127.19.84.3:48643)
0131 03:08:05.585450 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:05.585462 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:05.585516 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:05.585785 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:05.585786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:05.585789 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:05.586010 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:05.586012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.586751 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:05.586752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:05.587497 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:05.587499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.587559 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:05.587562 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:08:05.587587 (+ 25us) client_negotiation.cc:240] Negotiation successful
0131 03:08:05.587616 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260131 03:08:05.588209 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:05.585567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:48643 (local address 127.19.84.2:38183)
0131 03:08:05.585639 (+ 72us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:05.585641 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:05.585652 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:08:05.585686 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:05.585688 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:05.585704 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:05.585765 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:05.586086 (+ 321us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.586678 (+ 592us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:05.587578 (+ 900us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.588077 (+ 499us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:05.588086 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:05.588109 (+ 23us) server_negotiation.cc:300] Negotiation successful
0131 03:08:05.588140 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260131 03:08:05.588415 24982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:05.585420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:38781 (local address 127.19.84.3:39175)
0131 03:08:05.585672 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:05.585683 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:05.585751 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:05.585896 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:05.585897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:05.585898 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:05.586151 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:05.586152 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.586924 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:05.586925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:05.587612 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:05.587614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.588243 (+ 629us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:05.588247 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:08:05.588307 (+ 60us) client_negotiation.cc:240] Negotiation successful
0131 03:08:05.588349 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":10,"thread_start_us":96,"threads_started":1}
I20260131 03:08:05.588584 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:05.585461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:39175 (local address 127.19.84.1:38781)
0131 03:08:05.585511 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:05.585513 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:05.585771 (+ 258us) server_negotiation.cc:408] Connection header received
0131 03:08:05.585803 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:05.585804 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:05.585819 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:05.585889 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:05.586233 (+ 344us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.586831 (+ 598us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:05.587751 (+ 920us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.588205 (+ 454us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:05.588212 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:05.588471 (+ 259us) server_negotiation.cc:300] Negotiation successful
0131 03:08:05.588518 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260131 03:08:05.588568 24774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc9a2f0cc7aa4ed6b81e36e628e5ad7d" candidate_uuid: "d07875ff7c444ea5a4335f58f587cb07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7499984ffd8641f8bc4d0581fce45177" is_pre_election: true
I20260131 03:08:05.588732 24774 raft_consensus.cc:2468] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 7499984ffd8641f8bc4d0581fce45177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d07875ff7c444ea5a4335f58f587cb07 in term 0.
I20260131 03:08:05.588874 24639 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc9a2f0cc7aa4ed6b81e36e628e5ad7d" candidate_uuid: "d07875ff7c444ea5a4335f58f587cb07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c7430f8843742ddbf9e0bdfee5a779f" is_pre_election: true
I20260131 03:08:05.588972 24639 raft_consensus.cc:2468] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 6c7430f8843742ddbf9e0bdfee5a779f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d07875ff7c444ea5a4335f58f587cb07 in term 0.
I20260131 03:08:05.588992 24843 leader_election.cc:304] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7499984ffd8641f8bc4d0581fce45177, d07875ff7c444ea5a4335f58f587cb07; no voters:
I20260131 03:08:05.589098 24981 raft_consensus.cc:2804] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:08:05.589181 24981 raft_consensus.cc:493] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:08:05.589215 24981 raft_consensus.cc:3060] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:05.589864 24981 raft_consensus.cc:515] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7430f8843742ddbf9e0bdfee5a779f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38781 } } peers { permanent_uuid: "7499984ffd8641f8bc4d0581fce45177" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38183 } } peers { permanent_uuid: "d07875ff7c444ea5a4335f58f587cb07" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 40373 } }
I20260131 03:08:05.590036 24981 leader_election.cc:290] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07 [CANDIDATE]: Term 1 election: Requested vote from peers 6c7430f8843742ddbf9e0bdfee5a779f (127.19.84.1:38781), 7499984ffd8641f8bc4d0581fce45177 (127.19.84.2:38183)
I20260131 03:08:05.590221 24639 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc9a2f0cc7aa4ed6b81e36e628e5ad7d" candidate_uuid: "d07875ff7c444ea5a4335f58f587cb07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c7430f8843742ddbf9e0bdfee5a779f"
I20260131 03:08:05.590221 24774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fc9a2f0cc7aa4ed6b81e36e628e5ad7d" candidate_uuid: "d07875ff7c444ea5a4335f58f587cb07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7499984ffd8641f8bc4d0581fce45177"
I20260131 03:08:05.590305 24639 raft_consensus.cc:3060] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 6c7430f8843742ddbf9e0bdfee5a779f [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:05.590304 24774 raft_consensus.cc:3060] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 7499984ffd8641f8bc4d0581fce45177 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:05.591179 24774 raft_consensus.cc:2468] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 7499984ffd8641f8bc4d0581fce45177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d07875ff7c444ea5a4335f58f587cb07 in term 1.
I20260131 03:08:05.591179 24639 raft_consensus.cc:2468] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 6c7430f8843742ddbf9e0bdfee5a779f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d07875ff7c444ea5a4335f58f587cb07 in term 1.
I20260131 03:08:05.591409 24843 leader_election.cc:304] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7499984ffd8641f8bc4d0581fce45177, d07875ff7c444ea5a4335f58f587cb07; no voters:
I20260131 03:08:05.591536 24981 raft_consensus.cc:2804] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:08:05.591696 24981 raft_consensus.cc:697] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07 [term 1 LEADER]: Becoming Leader. State: Replica: d07875ff7c444ea5a4335f58f587cb07, State: Running, Role: LEADER
I20260131 03:08:05.591814 24981 consensus_queue.cc:237] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7430f8843742ddbf9e0bdfee5a779f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38781 } } peers { permanent_uuid: "7499984ffd8641f8bc4d0581fce45177" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38183 } } peers { permanent_uuid: "d07875ff7c444ea5a4335f58f587cb07" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 40373 } }
I20260131 03:08:05.592645 24492 catalog_manager.cc:5654] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07 reported cstate change: term changed from 0 to 1, leader changed from <none> to d07875ff7c444ea5a4335f58f587cb07 (127.19.84.3). New cstate: current_term: 1 leader_uuid: "d07875ff7c444ea5a4335f58f587cb07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7430f8843742ddbf9e0bdfee5a779f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7499984ffd8641f8bc4d0581fce45177" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d07875ff7c444ea5a4335f58f587cb07" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 40373 } health_report { overall_health: HEALTHY } } }
W20260131 03:08:05.612625 24957 tablet.cc:2378] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:08:05.619743 24972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:05.617339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51690 (local address 127.19.84.3:40373)
0131 03:08:05.617397 (+ 58us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:05.617399 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:05.617407 (+ 8us) server_negotiation.cc:408] Connection header received
0131 03:08:05.617463 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:05.617464 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:05.617481 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:05.617535 (+ 54us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:08:05.617817 (+ 282us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.618289 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:05.619012 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:05.619201 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:05.619249 (+ 48us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:08:05.619411 (+ 162us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:08:05.619486 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:05.619602 (+ 116us) server_negotiation.cc:300] Negotiation successful
0131 03:08:05.619627 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260131 03:08:05.621394 24774 raft_consensus.cc:1275] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 7499984ffd8641f8bc4d0581fce45177 [term 1 FOLLOWER]: Refusing update from remote peer d07875ff7c444ea5a4335f58f587cb07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:08:05.621409 24639 raft_consensus.cc:1275] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 6c7430f8843742ddbf9e0bdfee5a779f [term 1 FOLLOWER]: Refusing update from remote peer d07875ff7c444ea5a4335f58f587cb07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:08:05.621748 24983 consensus_queue.cc:1048] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7499984ffd8641f8bc4d0581fce45177" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:08:05.621866 24981 consensus_queue.cc:1048] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c7430f8843742ddbf9e0bdfee5a779f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:08:05.624132 24986 mvcc.cc:204] Tried to move back new op lower bound from 7249219115502354432 to 7249219115384651776. Current Snapshot: MvccSnapshot[applied={T|T < 7249219115502354432}]
I20260131 03:08:05.624653 24988 mvcc.cc:204] Tried to move back new op lower bound from 7249219115502354432 to 7249219115384651776. Current Snapshot: MvccSnapshot[applied={T|T < 7249219115502354432}]
I20260131 03:08:05.626447 24493 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:08:05.626544 24493 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:08:05.627323 24493 catalog_manager.cc:5941] T 00000000000000000000000000000000 P ec509b3c3a9540a184210589c0633103: Sending DeleteTablet for 3 replicas of tablet fc9a2f0cc7aa4ed6b81e36e628e5ad7d
I20260131 03:08:05.627676 24619 tablet_service.cc:1552] Processing DeleteTablet for tablet fc9a2f0cc7aa4ed6b81e36e628e5ad7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:05 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:51614
I20260131 03:08:05.627725 24754 tablet_service.cc:1552] Processing DeleteTablet for tablet fc9a2f0cc7aa4ed6b81e36e628e5ad7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:05 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:33344
I20260131 03:08:05.627733 24890 tablet_service.cc:1552] Processing DeleteTablet for tablet fc9a2f0cc7aa4ed6b81e36e628e5ad7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:05 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:51688
I20260131 03:08:05.627951 24997 tablet_replica.cc:333] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 7499984ffd8641f8bc4d0581fce45177: stopping tablet replica
I20260131 03:08:05.628023 24996 tablet_replica.cc:333] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 6c7430f8843742ddbf9e0bdfee5a779f: stopping tablet replica
I20260131 03:08:05.628067 24998 tablet_replica.cc:333] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07: stopping tablet replica
I20260131 03:08:05.628134 24996 raft_consensus.cc:2243] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 6c7430f8843742ddbf9e0bdfee5a779f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:08:05.628139 24998 raft_consensus.cc:2243] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:08:05.628230 24998 raft_consensus.cc:2272] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:08:05.628322 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 24557
I20260131 03:08:05.628337 24996 raft_consensus.cc:2272] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 6c7430f8843742ddbf9e0bdfee5a779f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:08:05.628351 24997 raft_consensus.cc:2243] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 7499984ffd8641f8bc4d0581fce45177 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:08:05.629117 24997 raft_consensus.cc:2272] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 7499984ffd8641f8bc4d0581fce45177 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:08:05.629549 24998 ts_tablet_manager.cc:1916] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:08:05.630860 24997 ts_tablet_manager.cc:1916] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 7499984ffd8641f8bc4d0581fce45177: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:08:05.631060 24998 ts_tablet_manager.cc:1929] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:08:05.631124 24998 log.cc:1199] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/fc9a2f0cc7aa4ed6b81e36e628e5ad7d
I20260131 03:08:05.631408 24998 ts_tablet_manager.cc:1950] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P d07875ff7c444ea5a4335f58f587cb07: Deleting consensus metadata
I20260131 03:08:05.632186 24477 catalog_manager.cc:4985] TS d07875ff7c444ea5a4335f58f587cb07 (127.19.84.3:40373): tablet fc9a2f0cc7aa4ed6b81e36e628e5ad7d (table test-table [id=2e9491b1b17c437ca5cdb51b07bfa13f]) successfully deleted
I20260131 03:08:05.632404 24997 ts_tablet_manager.cc:1929] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 7499984ffd8641f8bc4d0581fce45177: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:08:05.632484 24997 log.cc:1199] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 7499984ffd8641f8bc4d0581fce45177: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769828849314684-19792-0/minicluster-data/ts-1/wal/wals/fc9a2f0cc7aa4ed6b81e36e628e5ad7d
I20260131 03:08:05.632982 24997 ts_tablet_manager.cc:1950] T fc9a2f0cc7aa4ed6b81e36e628e5ad7d P 7499984ffd8641f8bc4d0581fce45177: Deleting consensus metadata
I20260131 03:08:05.633363 24478 catalog_manager.cc:4985] TS 7499984ffd8641f8bc4d0581fce45177 (127.19.84.2:38183): tablet fc9a2f0cc7aa4ed6b81e36e628e5ad7d (table test-table [id=2e9491b1b17c437ca5cdb51b07bfa13f]) successfully deleted
W20260131 03:08:05.636466 24477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.84.1:38781 (error 108)
I20260131 03:08:05.636750 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 24692
I20260131 03:08:05.637471 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:05.636899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:38781 (local address 127.0.0.1:51622)
0131 03:08:05.637395 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:05.637430 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.1:38781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476}
W20260131 03:08:05.637902 24477 catalog_manager.cc:4712] TS 6c7430f8843742ddbf9e0bdfee5a779f (127.19.84.1:38781): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc9a2f0cc7aa4ed6b81e36e628e5ad7d: Network error: Client connection negotiation failed: client connection to 127.19.84.1:38781: connect: Connection refused (error 111)
I20260131 03:08:05.641814 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 24827
I20260131 03:08:05.646662 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 24462
2026-01-31T03:08:05Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3151 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-01-31T03:08:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:08:05Z Disabled control of system clock
I20260131 03:08:05.673686 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:44945
--webserver_interface=127.19.84.62
--webserver_port=0
--builtin_ntp_servers=127.19.84.20:36849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:44945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260131 03:08:05.753414 25003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:05.753633 25003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:05.753656 25003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:05.755290 25003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:08:05.755352 25003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:05.755364 25003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:05.755407 25003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:08:05.755420 25003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:08:05.757195 25003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:36849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:44945
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:44945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:05.757472 25003 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:05.757742 25003 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:05.760396 25009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:05.760603 25003 server_base.cc:1060] running on GCE node
W20260131 03:08:05.760396 25011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:05.760464 25008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:05.761204 25003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:05.761483 25003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:05.762705 25003 hybrid_clock.cc:648] HybridClock initialized: now 1769828885762678 us; error 40 us; skew 500 ppm
I20260131 03:08:05.764003 25003 webserver.cc:492] Webserver started at http://127.19.84.62:35195/ using document root <none> and password file <none>
I20260131 03:08:05.764225 25003 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:05.764288 25003 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:05.764434 25003 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:05.765419 25003 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "e3bb3e394154486a9ec114340190f2cd"
format_stamp: "Formatted at 2026-01-31 03:08:05 on dist-test-slave-5nnp"
server_key: "b4a77987a193545cc46247aa1a7289f5"
server_key_iv: "04637ba5b7a0db5db25f68d2535d9b43"
server_key_version: "encryptionkey@0"
I20260131 03:08:05.765796 25003 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "e3bb3e394154486a9ec114340190f2cd"
format_stamp: "Formatted at 2026-01-31 03:08:05 on dist-test-slave-5nnp"
server_key: "b4a77987a193545cc46247aa1a7289f5"
server_key_iv: "04637ba5b7a0db5db25f68d2535d9b43"
server_key_version: "encryptionkey@0"
I20260131 03:08:05.767028 25003 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260131 03:08:05.767930 25017 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:05.768122 25003 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260131 03:08:05.768213 25003 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "e3bb3e394154486a9ec114340190f2cd"
format_stamp: "Formatted at 2026-01-31 03:08:05 on dist-test-slave-5nnp"
server_key: "b4a77987a193545cc46247aa1a7289f5"
server_key_iv: "04637ba5b7a0db5db25f68d2535d9b43"
server_key_version: "encryptionkey@0"
I20260131 03:08:05.768294 25003 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:05.771890 25003 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:05.772135 25003 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:05.772261 25003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:05.776032 25003 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:44945
I20260131 03:08:05.776077 25069 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:44945 every 8 connection(s)
I20260131 03:08:05.776412 25003 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:08:05.777017 25070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:05.777592 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 25003
I20260131 03:08:05.777729 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:08:05.777966 19792 external_mini_cluster.cc:1468] Setting key 9e8d53ad8bb97e76ee486d803058a3df
I20260131 03:08:05.780100 25070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd: Bootstrap starting.
I20260131 03:08:05.780589 25073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:05.778531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40756 (local address 127.19.84.62:44945)
0131 03:08:05.778744 (+ 213us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:05.778753 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:05.778817 (+ 64us) server_negotiation.cc:408] Connection header received
0131 03:08:05.779225 (+ 408us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:05.779232 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:05.779278 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:05.779448 (+ 170us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:05.780146 (+ 698us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:05.780149 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:05.780150 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:08:05.780155 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:05.780189 (+ 34us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:08:05.780196 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:05.780267 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:05.780364 (+ 97us) server_negotiation.cc:300] Negotiation successful
0131 03:08:05.780416 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
I20260131 03:08:05.780823 25070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:05.781086 25070 log.cc:826] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:05.781831 25070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd: No bootstrap required, opened a new log
I20260131 03:08:05.782892 25070 raft_consensus.cc:359] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bb3e394154486a9ec114340190f2cd" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44945 } }
I20260131 03:08:05.783021 25070 raft_consensus.cc:385] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:05.783041 25070 raft_consensus.cc:740] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3bb3e394154486a9ec114340190f2cd, State: Initialized, Role: FOLLOWER
I20260131 03:08:05.783157 25070 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bb3e394154486a9ec114340190f2cd" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44945 } }
I20260131 03:08:05.783232 25070 raft_consensus.cc:399] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:08:05.783257 25070 raft_consensus.cc:493] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:08:05.783285 25070 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:05.783896 25070 raft_consensus.cc:515] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bb3e394154486a9ec114340190f2cd" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44945 } }
I20260131 03:08:05.783990 25070 leader_election.cc:304] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3bb3e394154486a9ec114340190f2cd; no voters:
I20260131 03:08:05.784138 25070 leader_election.cc:290] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:08:05.784186 25075 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:08:05.784291 25075 raft_consensus.cc:697] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd [term 1 LEADER]: Becoming Leader. State: Replica: e3bb3e394154486a9ec114340190f2cd, State: Running, Role: LEADER
I20260131 03:08:05.784374 25070 sys_catalog.cc:565] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:08:05.784366 25075 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bb3e394154486a9ec114340190f2cd" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44945 } }
I20260131 03:08:05.784874 25076 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3bb3e394154486a9ec114340190f2cd. Latest consensus state: current_term: 1 leader_uuid: "e3bb3e394154486a9ec114340190f2cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bb3e394154486a9ec114340190f2cd" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44945 } } }
I20260131 03:08:05.784906 25077 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3bb3e394154486a9ec114340190f2cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3bb3e394154486a9ec114340190f2cd" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44945 } } }
I20260131 03:08:05.785066 25077 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:05.785261 25076 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd [sys.catalog]: This master's current role is: LEADER
W20260131 03:08:05.786480 25091 catalog_manager.cc:1568] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:08:05.786594 25091 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:08:05.786659 25084 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:08:05.786810 25084 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:08:05.788269 25084 catalog_manager.cc:1357] Generated new cluster ID: fdf17b54ebf04ae18378678974871702
I20260131 03:08:05.788332 25084 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:08:05.828701 25084 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:08:05.828894 25084 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:08:05.837446 25084 catalog_manager.cc:6027] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd: Generated new TSK 0
I20260131 03:08:05.837702 25084 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:08:05.845822 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:44945
--builtin_ntp_servers=127.19.84.20:36849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260131 03:08:05.926856 25094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:05.927033 25094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:05.927057 25094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:05.928638 25094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:05.928694 25094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:05.928720 25094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:08:05.930392 25094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:36849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:05.930634 25094 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:05.930909 25094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:05.933991 25099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:05.933977 25100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:05.933977 25102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:05.934405 25094 server_base.cc:1060] running on GCE node
I20260131 03:08:05.934746 25094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:05.934994 25094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:05.936172 25094 hybrid_clock.cc:648] HybridClock initialized: now 1769828885936156 us; error 32 us; skew 500 ppm
I20260131 03:08:05.937439 25094 webserver.cc:492] Webserver started at http://127.19.84.1:37395/ using document root <none> and password file <none>
I20260131 03:08:05.937681 25094 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:05.937747 25094 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:05.937894 25094 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:05.938925 25094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "fe9b6ef2dce04a1982a805f3f3896a92"
format_stamp: "Formatted at 2026-01-31 03:08:05 on dist-test-slave-5nnp"
server_key: "c3071909c82b090a5de3df7d458caf42"
server_key_iv: "55000d4f79a4961369bdf39686bedf8b"
server_key_version: "encryptionkey@0"
I20260131 03:08:05.939240 25094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "fe9b6ef2dce04a1982a805f3f3896a92"
format_stamp: "Formatted at 2026-01-31 03:08:05 on dist-test-slave-5nnp"
server_key: "c3071909c82b090a5de3df7d458caf42"
server_key_iv: "55000d4f79a4961369bdf39686bedf8b"
server_key_version: "encryptionkey@0"
I20260131 03:08:05.940649 25094 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:08:05.941465 25108 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:05.941697 25094 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:08:05.941792 25094 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "fe9b6ef2dce04a1982a805f3f3896a92"
format_stamp: "Formatted at 2026-01-31 03:08:05 on dist-test-slave-5nnp"
server_key: "c3071909c82b090a5de3df7d458caf42"
server_key_iv: "55000d4f79a4961369bdf39686bedf8b"
server_key_version: "encryptionkey@0"
I20260131 03:08:05.941882 25094 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:05.945470 25094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:05.945698 25094 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:05.945813 25094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:05.946025 25094 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:05.946321 25094 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:05.946372 25094 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:05.946414 25094 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:05.946448 25094 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:05.953373 25094 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:38737
I20260131 03:08:05.953438 25221 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:38737 every 8 connection(s)
I20260131 03:08:05.953785 25094 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
I20260131 03:08:05.956959 25073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:05.954622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:46531 (local address 127.19.84.62:44945)
0131 03:08:05.954714 (+ 92us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:05.954717 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:05.955052 (+ 335us) server_negotiation.cc:408] Connection header received
0131 03:08:05.955816 (+ 764us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:05.955818 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:05.955826 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:05.955882 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:05.956698 (+ 816us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:05.956700 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:05.956700 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:08:05.956704 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:05.956721 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:08:05.956724 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:05.956781 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:05.956886 (+ 105us) server_negotiation.cc:300] Negotiation successful
0131 03:08:05.956900 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260131 03:08:05.957180 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:05.954683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44945 (local address 127.19.84.1:46531)
0131 03:08:05.954916 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:05.954949 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:05.955659 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:05.955951 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:05.955956 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:05.955968 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:05.956543 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:08:05.956543 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:05.956569 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:08:05.956592 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:05.956795 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:05.956800 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:05.956803 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:08:05.956904 (+ 101us) client_negotiation.cc:240] Negotiation successful
0131 03:08:05.956956 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20260131 03:08:05.957640 25222 heartbeater.cc:344] Connected to a master server at 127.19.84.62:44945
I20260131 03:08:05.957756 25222 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:05.957914 25222 heartbeater.cc:507] Master 127.19.84.62:44945 requested a full tablet report, sending...
I20260131 03:08:05.958402 25034 ts_manager.cc:194] Registered new tserver with Master: fe9b6ef2dce04a1982a805f3f3896a92 (127.19.84.1:38737)
I20260131 03:08:05.960356 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 25094
I20260131 03:08:05.960469 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:08:05.960650 19792 external_mini_cluster.cc:1468] Setting key e92d3323e201232077c9f5576fa68568
I20260131 03:08:05.961666 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:44945
--builtin_ntp_servers=127.19.84.20:36849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260131 03:08:06.044823 25225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:06.045038 25225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:06.045074 25225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:06.046651 25225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:06.046715 25225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:06.046756 25225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:08:06.048373 25225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:36849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:06.048633 25225 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:06.048882 25225 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:06.051600 25231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:06.051573 25233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:06.051570 25230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:06.051916 25225 server_base.cc:1060] running on GCE node
I20260131 03:08:06.052098 25225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:06.052337 25225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:06.053512 25225 hybrid_clock.cc:648] HybridClock initialized: now 1769828886053490 us; error 39 us; skew 500 ppm
I20260131 03:08:06.054939 25225 webserver.cc:492] Webserver started at http://127.19.84.2:45775/ using document root <none> and password file <none>
I20260131 03:08:06.055171 25225 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:06.055236 25225 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:06.055435 25225 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:06.056469 25225 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "2ebbad9d6d24444cbc16f88fbde32079"
format_stamp: "Formatted at 2026-01-31 03:08:06 on dist-test-slave-5nnp"
server_key: "f251a3fd00abf39efd2966a3a997182b"
server_key_iv: "a51f38cb7a4738c98ed05c7d7e626bb5"
server_key_version: "encryptionkey@0"
I20260131 03:08:06.056828 25225 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "2ebbad9d6d24444cbc16f88fbde32079"
format_stamp: "Formatted at 2026-01-31 03:08:06 on dist-test-slave-5nnp"
server_key: "f251a3fd00abf39efd2966a3a997182b"
server_key_iv: "a51f38cb7a4738c98ed05c7d7e626bb5"
server_key_version: "encryptionkey@0"
I20260131 03:08:06.058099 25225 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260131 03:08:06.059088 25239 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:06.059475 25225 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:08:06.059574 25225 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "2ebbad9d6d24444cbc16f88fbde32079"
format_stamp: "Formatted at 2026-01-31 03:08:06 on dist-test-slave-5nnp"
server_key: "f251a3fd00abf39efd2966a3a997182b"
server_key_iv: "a51f38cb7a4738c98ed05c7d7e626bb5"
server_key_version: "encryptionkey@0"
I20260131 03:08:06.059665 25225 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:06.063660 25225 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:06.064015 25225 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:06.064153 25225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:06.064477 25225 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:06.065001 25225 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:06.065058 25225 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:06.065100 25225 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:06.065129 25225 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:06.071914 25225 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:35437
I20260131 03:08:06.071985 25352 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:35437 every 8 connection(s)
I20260131 03:08:06.072358 25225 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
I20260131 03:08:06.075054 25073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:06.073099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:37925 (local address 127.19.84.62:44945)
0131 03:08:06.073149 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:06.073151 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:06.073441 (+ 290us) server_negotiation.cc:408] Connection header received
0131 03:08:06.073948 (+ 507us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:06.073949 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:06.073956 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:06.074017 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:06.074827 (+ 810us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:06.074828 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:06.074829 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:08:06.074832 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:06.074848 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:08:06.074851 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:06.074899 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:06.074984 (+ 85us) server_negotiation.cc:300] Negotiation successful
0131 03:08:06.074999 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260131 03:08:06.075205 25355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:06.073080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44945 (local address 127.19.84.2:37925)
0131 03:08:06.073340 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:06.073376 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:06.073844 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:06.074084 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:06.074087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:06.074096 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:06.074678 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:08:06.074678 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:06.074699 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:08:06.074717 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:06.074904 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:06.074906 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:06.074909 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:08:06.074976 (+ 67us) client_negotiation.cc:240] Negotiation successful
0131 03:08:06.075014 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":93,"threads_started":1}
I20260131 03:08:06.075747 25353 heartbeater.cc:344] Connected to a master server at 127.19.84.62:44945
I20260131 03:08:06.075843 25353 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:06.076023 25353 heartbeater.cc:507] Master 127.19.84.62:44945 requested a full tablet report, sending...
I20260131 03:08:06.076337 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 25225
I20260131 03:08:06.076431 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:08:06.076434 25034 ts_manager.cc:194] Registered new tserver with Master: 2ebbad9d6d24444cbc16f88fbde32079 (127.19.84.2:35437)
I20260131 03:08:06.076653 19792 external_mini_cluster.cc:1468] Setting key d87b89d72a81d9b4d7034c8983bd3201
I20260131 03:08:06.077632 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:44945
--builtin_ntp_servers=127.19.84.20:36849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260131 03:08:06.161286 25356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:06.161469 25356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:06.161492 25356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:06.163048 25356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:06.163095 25356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:06.163122 25356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:08:06.164836 25356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:36849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:44945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:06.165112 25356 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:06.165349 25356 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:06.168357 25364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:06.168370 25361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:06.168534 25356 server_base.cc:1060] running on GCE node
W20260131 03:08:06.168359 25362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:06.168826 25356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:06.169224 25356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:06.170388 25356 hybrid_clock.cc:648] HybridClock initialized: now 1769828886170372 us; error 40 us; skew 500 ppm
I20260131 03:08:06.171736 25356 webserver.cc:492] Webserver started at http://127.19.84.3:35093/ using document root <none> and password file <none>
I20260131 03:08:06.171995 25356 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:06.172058 25356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:06.172191 25356 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:06.173151 25356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "7abfc0422bf24c908a27d8e854be330d"
format_stamp: "Formatted at 2026-01-31 03:08:06 on dist-test-slave-5nnp"
server_key: "dbbe5f4530c27283ab730c3a48ecc1d7"
server_key_iv: "59d24c75ca12cc3cfaef3d9d4a2eba7c"
server_key_version: "encryptionkey@0"
I20260131 03:08:06.173489 25356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "7abfc0422bf24c908a27d8e854be330d"
format_stamp: "Formatted at 2026-01-31 03:08:06 on dist-test-slave-5nnp"
server_key: "dbbe5f4530c27283ab730c3a48ecc1d7"
server_key_iv: "59d24c75ca12cc3cfaef3d9d4a2eba7c"
server_key_version: "encryptionkey@0"
I20260131 03:08:06.174719 25356 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260131 03:08:06.175597 25370 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:06.175813 25356 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:08:06.175915 25356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "7abfc0422bf24c908a27d8e854be330d"
format_stamp: "Formatted at 2026-01-31 03:08:06 on dist-test-slave-5nnp"
server_key: "dbbe5f4530c27283ab730c3a48ecc1d7"
server_key_iv: "59d24c75ca12cc3cfaef3d9d4a2eba7c"
server_key_version: "encryptionkey@0"
I20260131 03:08:06.175997 25356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:06.179507 25356 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:06.179749 25356 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:06.179876 25356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:06.180089 25356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:06.180390 25356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:06.180442 25356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:06.180482 25356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:06.180513 25356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:06.187032 25356 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:34555
I20260131 03:08:06.187079 25483 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:34555 every 8 connection(s)
I20260131 03:08:06.187475 25356 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
I20260131 03:08:06.190261 25073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:06.188240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:33577 (local address 127.19.84.62:44945)
0131 03:08:06.188307 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:06.188309 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:06.188529 (+ 220us) server_negotiation.cc:408] Connection header received
0131 03:08:06.189043 (+ 514us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:06.189044 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:06.189049 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:06.189120 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:06.189958 (+ 838us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:06.189960 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:06.189960 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:08:06.189965 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:06.189980 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:08:06.189984 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:06.190039 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:06.190192 (+ 153us) server_negotiation.cc:300] Negotiation successful
0131 03:08:06.190207 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260131 03:08:06.190416 25486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:06.188209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44945 (local address 127.19.84.3:33577)
0131 03:08:06.188400 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:06.188436 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:06.188880 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:06.189224 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:06.189228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:06.189237 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:06.189797 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:08:06.189798 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:06.189821 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:08:06.189840 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:06.190077 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:06.190082 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:06.190085 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:08:06.190180 (+ 95us) client_negotiation.cc:240] Negotiation successful
0131 03:08:06.190225 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":93,"threads_started":1}
I20260131 03:08:06.191021 25484 heartbeater.cc:344] Connected to a master server at 127.19.84.62:44945
I20260131 03:08:06.191134 25484 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:06.191272 25484 heartbeater.cc:507] Master 127.19.84.62:44945 requested a full tablet report, sending...
I20260131 03:08:06.191708 25034 ts_manager.cc:194] Registered new tserver with Master: 7abfc0422bf24c908a27d8e854be330d (127.19.84.3:34555)
I20260131 03:08:06.192557 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 25356
I20260131 03:08:06.192648 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:08:06.192783 19792 external_mini_cluster.cc:1468] Setting key f194756f1ae858a98159261062c6ebfd
I20260131 03:08:06.193867 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:08:06.196310 25073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:06.195170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40768 (local address 127.19.84.62:44945)
0131 03:08:06.195231 (+ 61us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:06.195233 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:06.195367 (+ 134us) server_negotiation.cc:408] Connection header received
0131 03:08:06.195451 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:06.195452 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:06.195457 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:06.195501 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:06.196126 (+ 625us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:06.196127 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:06.196127 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:08:06.196131 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:06.196145 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:08:06.196148 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:06.196201 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:06.196244 (+ 43us) server_negotiation.cc:300] Negotiation successful
0131 03:08:06.196257 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260131 03:08:06.197798 25034 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:08:06.199149 25034 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:08:06.203414 25492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:06.201365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:34555 (local address 127.0.0.1:48874)
0131 03:08:06.201515 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:06.201545 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:06.201592 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:06.202119 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:06.202120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:06.202123 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:06.202709 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:08:06.202710 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:06.202736 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:08:06.202759 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:06.203161 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:06.203167 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:06.203170 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:08:06.203258 (+ 88us) client_negotiation.cc:240] Negotiation successful
0131 03:08:06.203276 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":103,"threads_started":1}
I20260131 03:08:06.203442 25497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:06.201787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48874 (local address 127.19.84.3:34555)
0131 03:08:06.201959 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:06.201976 (+ 17us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:06.201986 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:08:06.202013 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:06.202015 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:06.202021 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:06.202323 (+ 302us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:06.203029 (+ 706us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:06.203032 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:06.203033 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:08:06.203037 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:06.203077 (+ 40us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:08:06.203086 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:06.203163 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:06.203279 (+ 116us) server_negotiation.cc:300] Negotiation successful
0131 03:08:06.203294 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260131 03:08:06.203595 25496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:06.201760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42320 (local address 127.19.84.2:35437)
0131 03:08:06.201970 (+ 210us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:06.201977 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:06.201986 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:08:06.202012 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:06.202014 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:06.202021 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:06.202401 (+ 380us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:06.203191 (+ 790us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:06.203193 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:06.203194 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:08:06.203197 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:06.203256 (+ 59us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:08:06.203263 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:06.203469 (+ 206us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:06.203492 (+ 23us) server_negotiation.cc:300] Negotiation successful
0131 03:08:06.203508 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":101,"threads_started":1}
I20260131 03:08:06.203806 25495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:06.201611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:35437 (local address 127.0.0.1:42320)
0131 03:08:06.201754 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:06.201764 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:06.201809 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:06.202269 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:06.202270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:06.202272 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:06.202966 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:08:06.202967 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:06.202975 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:08:06.202986 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:06.203345 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:06.203346 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:06.203347 (+ 1us) client_negotiation.cc:769] Sending connection context
0131 03:08:06.203376 (+ 29us) client_negotiation.cc:240] Negotiation successful
0131 03:08:06.203411 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
I20260131 03:08:06.204291 25493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:06.201421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:38737 (local address 127.0.0.1:54586)
0131 03:08:06.201726 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:06.201738 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:06.201792 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:06.202086 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:06.202090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:06.202096 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:06.202845 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:08:06.202845 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:06.202851 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:08:06.202860 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:06.204145 (+ 1285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:06.204146 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:06.204147 (+ 1us) client_negotiation.cc:769] Sending connection context
0131 03:08:06.204219 (+ 72us) client_negotiation.cc:240] Negotiation successful
0131 03:08:06.204234 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":96,"threads_started":1}
I20260131 03:08:06.204288 25418 tablet_service.cc:1505] Processing CreateTablet for tablet e8e59dc4abdc402c93ea908d88361216 (DEFAULT_TABLE table=test-table [id=0f4c5a27b2fb411b95a0cfdbfe2b3173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:08:06.204332 25287 tablet_service.cc:1505] Processing CreateTablet for tablet e8e59dc4abdc402c93ea908d88361216 (DEFAULT_TABLE table=test-table [id=0f4c5a27b2fb411b95a0cfdbfe2b3173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:08:06.204550 25494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:06.201522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54586 (local address 127.19.84.1:38737)
0131 03:08:06.201956 (+ 434us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:06.201962 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:06.201971 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:08:06.201993 (+ 22us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:06.201995 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:06.202001 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:06.202085 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:06.204018 (+ 1933us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:06.204021 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:06.204021 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:08:06.204025 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:06.204047 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:08:06.204053 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:06.204119 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:06.204245 (+ 126us) server_negotiation.cc:300] Negotiation successful
0131 03:08:06.204260 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":120,"threads_started":1}
I20260131 03:08:06.204605 25287 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8e59dc4abdc402c93ea908d88361216. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:06.204605 25418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8e59dc4abdc402c93ea908d88361216. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:06.205173 25156 tablet_service.cc:1505] Processing CreateTablet for tablet e8e59dc4abdc402c93ea908d88361216 (DEFAULT_TABLE table=test-table [id=0f4c5a27b2fb411b95a0cfdbfe2b3173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:08:06.205461 25156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8e59dc4abdc402c93ea908d88361216. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:06.207442 25498 tablet_bootstrap.cc:492] T e8e59dc4abdc402c93ea908d88361216 P 7abfc0422bf24c908a27d8e854be330d: Bootstrap starting.
I20260131 03:08:06.207748 25500 tablet_bootstrap.cc:492] T e8e59dc4abdc402c93ea908d88361216 P fe9b6ef2dce04a1982a805f3f3896a92: Bootstrap starting.
I20260131 03:08:06.208213 25498 tablet_bootstrap.cc:654] T e8e59dc4abdc402c93ea908d88361216 P 7abfc0422bf24c908a27d8e854be330d: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:06.208271 25500 tablet_bootstrap.cc:654] T e8e59dc4abdc402c93ea908d88361216 P fe9b6ef2dce04a1982a805f3f3896a92: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:06.208482 25498 log.cc:826] T e8e59dc4abdc402c93ea908d88361216 P 7abfc0422bf24c908a27d8e854be330d: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:06.208521 25500 log.cc:826] T e8e59dc4abdc402c93ea908d88361216 P fe9b6ef2dce04a1982a805f3f3896a92: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:06.208818 25499 tablet_bootstrap.cc:492] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079: Bootstrap starting.
I20260131 03:08:06.209199 25500 tablet_bootstrap.cc:492] T e8e59dc4abdc402c93ea908d88361216 P fe9b6ef2dce04a1982a805f3f3896a92: No bootstrap required, opened a new log
I20260131 03:08:06.209218 25498 tablet_bootstrap.cc:492] T e8e59dc4abdc402c93ea908d88361216 P 7abfc0422bf24c908a27d8e854be330d: No bootstrap required, opened a new log
I20260131 03:08:06.209280 25498 ts_tablet_manager.cc:1403] T e8e59dc4abdc402c93ea908d88361216 P 7abfc0422bf24c908a27d8e854be330d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:08:06.209282 25500 ts_tablet_manager.cc:1403] T e8e59dc4abdc402c93ea908d88361216 P fe9b6ef2dce04a1982a805f3f3896a92: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:08:06.209395 25499 tablet_bootstrap.cc:654] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:06.209664 25499 log.cc:826] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:06.210497 25499 tablet_bootstrap.cc:492] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079: No bootstrap required, opened a new log
I20260131 03:08:06.210567 25499 ts_tablet_manager.cc:1403] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:08:06.210923 25500 raft_consensus.cc:359] T e8e59dc4abdc402c93ea908d88361216 P fe9b6ef2dce04a1982a805f3f3896a92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abfc0422bf24c908a27d8e854be330d" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 34555 } } peers { permanent_uuid: "2ebbad9d6d24444cbc16f88fbde32079" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 35437 } } peers { permanent_uuid: "fe9b6ef2dce04a1982a805f3f3896a92" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38737 } }
I20260131 03:08:06.211072 25500 raft_consensus.cc:385] T e8e59dc4abdc402c93ea908d88361216 P fe9b6ef2dce04a1982a805f3f3896a92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:06.211113 25500 raft_consensus.cc:740] T e8e59dc4abdc402c93ea908d88361216 P fe9b6ef2dce04a1982a805f3f3896a92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe9b6ef2dce04a1982a805f3f3896a92, State: Initialized, Role: FOLLOWER
I20260131 03:08:06.211019 25498 raft_consensus.cc:359] T e8e59dc4abdc402c93ea908d88361216 P 7abfc0422bf24c908a27d8e854be330d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abfc0422bf24c908a27d8e854be330d" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 34555 } } peers { permanent_uuid: "2ebbad9d6d24444cbc16f88fbde32079" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 35437 } } peers { permanent_uuid: "fe9b6ef2dce04a1982a805f3f3896a92" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38737 } }
I20260131 03:08:06.211140 25498 raft_consensus.cc:385] T e8e59dc4abdc402c93ea908d88361216 P 7abfc0422bf24c908a27d8e854be330d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:06.211166 25498 raft_consensus.cc:740] T e8e59dc4abdc402c93ea908d88361216 P 7abfc0422bf24c908a27d8e854be330d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7abfc0422bf24c908a27d8e854be330d, State: Initialized, Role: FOLLOWER
I20260131 03:08:06.211221 25500 consensus_queue.cc:260] T e8e59dc4abdc402c93ea908d88361216 P fe9b6ef2dce04a1982a805f3f3896a92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abfc0422bf24c908a27d8e854be330d" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 34555 } } peers { permanent_uuid: "2ebbad9d6d24444cbc16f88fbde32079" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 35437 } } peers { permanent_uuid: "fe9b6ef2dce04a1982a805f3f3896a92" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38737 } }
I20260131 03:08:06.211273 25498 consensus_queue.cc:260] T e8e59dc4abdc402c93ea908d88361216 P 7abfc0422bf24c908a27d8e854be330d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abfc0422bf24c908a27d8e854be330d" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 34555 } } peers { permanent_uuid: "2ebbad9d6d24444cbc16f88fbde32079" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 35437 } } peers { permanent_uuid: "fe9b6ef2dce04a1982a805f3f3896a92" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38737 } }
I20260131 03:08:06.211521 25500 ts_tablet_manager.cc:1434] T e8e59dc4abdc402c93ea908d88361216 P fe9b6ef2dce04a1982a805f3f3896a92: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:08:06.211536 25498 ts_tablet_manager.cc:1434] T e8e59dc4abdc402c93ea908d88361216 P 7abfc0422bf24c908a27d8e854be330d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:08:06.211563 25484 heartbeater.cc:499] Master 127.19.84.62:44945 was elected leader, sending a full tablet report...
I20260131 03:08:06.211663 25222 heartbeater.cc:499] Master 127.19.84.62:44945 was elected leader, sending a full tablet report...
I20260131 03:08:06.212219 25499 raft_consensus.cc:359] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abfc0422bf24c908a27d8e854be330d" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 34555 } } peers { permanent_uuid: "2ebbad9d6d24444cbc16f88fbde32079" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 35437 } } peers { permanent_uuid: "fe9b6ef2dce04a1982a805f3f3896a92" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38737 } }
I20260131 03:08:06.212356 25499 raft_consensus.cc:385] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:06.212385 25499 raft_consensus.cc:740] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ebbad9d6d24444cbc16f88fbde32079, State: Initialized, Role: FOLLOWER
I20260131 03:08:06.212491 25499 consensus_queue.cc:260] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abfc0422bf24c908a27d8e854be330d" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 34555 } } peers { permanent_uuid: "2ebbad9d6d24444cbc16f88fbde32079" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 35437 } } peers { permanent_uuid: "fe9b6ef2dce04a1982a805f3f3896a92" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38737 } }
I20260131 03:08:06.212738 25499 ts_tablet_manager.cc:1434] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:08:06.212829 25353 heartbeater.cc:499] Master 127.19.84.62:44945 was elected leader, sending a full tablet report...
W20260131 03:08:06.323148 25354 tablet.cc:2378] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:08:06.400765 25507 raft_consensus.cc:493] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:08:06.400918 25507 raft_consensus.cc:515] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abfc0422bf24c908a27d8e854be330d" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 34555 } } peers { permanent_uuid: "2ebbad9d6d24444cbc16f88fbde32079" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 35437 } } peers { permanent_uuid: "fe9b6ef2dce04a1982a805f3f3896a92" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38737 } }
I20260131 03:08:06.401275 25507 leader_election.cc:290] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7abfc0422bf24c908a27d8e854be330d (127.19.84.3:34555), fe9b6ef2dce04a1982a805f3f3896a92 (127.19.84.1:38737)
I20260131 03:08:06.402565 25497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:06.401556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:42515 (local address 127.19.84.3:34555)
0131 03:08:06.401615 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:06.401617 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:06.401629 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:08:06.401661 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:06.401662 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:06.401668 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:06.401739 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:06.402341 (+ 602us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:06.402343 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:06.402343 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:08:06.402348 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:06.402367 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:08:06.402375 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:06.402420 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:06.402495 (+ 75us) server_negotiation.cc:300] Negotiation successful
0131 03:08:06.402510 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260131 03:08:06.402580 25355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:06.401470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:34555 (local address 127.19.84.2:42515)
0131 03:08:06.401543 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:06.401556 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:06.401596 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:06.401760 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:06.401761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:06.401763 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:06.402209 (+ 446us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:08:06.402209 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:06.402215 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:08:06.402226 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:06.402432 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:06.402433 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:06.402433 (+ 0us) client_negotiation.cc:769] Sending connection context
0131 03:08:06.402482 (+ 49us) client_negotiation.cc:240] Negotiation successful
0131 03:08:06.402498 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260131 03:08:06.402993 25438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8e59dc4abdc402c93ea908d88361216" candidate_uuid: "2ebbad9d6d24444cbc16f88fbde32079" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7abfc0422bf24c908a27d8e854be330d" is_pre_election: true
I20260131 03:08:06.403158 25438 raft_consensus.cc:2468] T e8e59dc4abdc402c93ea908d88361216 P 7abfc0422bf24c908a27d8e854be330d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ebbad9d6d24444cbc16f88fbde32079 in term 0.
I20260131 03:08:06.403165 25494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:06.401600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:38403 (local address 127.19.84.1:38737)
0131 03:08:06.401658 (+ 58us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:06.401660 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:06.401828 (+ 168us) server_negotiation.cc:408] Connection header received
0131 03:08:06.401853 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:06.401854 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:06.401859 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:06.401900 (+ 41us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:06.402941 (+ 1041us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:06.402942 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:06.402942 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:08:06.402947 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:06.402963 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:08:06.402967 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:06.403024 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:06.403083 (+ 59us) server_negotiation.cc:300] Negotiation successful
0131 03:08:06.403100 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260131 03:08:06.403185 25508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:06.401524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:38737 (local address 127.19.84.2:38403)
0131 03:08:06.401706 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:06.401733 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:06.401785 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:06.401935 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:06.401936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:06.401938 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:06.402835 (+ 897us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:08:06.402836 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:06.402842 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:08:06.402853 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:06.403027 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:06.403028 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:06.403029 (+ 1us) client_negotiation.cc:769] Sending connection context
0131 03:08:06.403082 (+ 53us) client_negotiation.cc:240] Negotiation successful
0131 03:08:06.403097 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":125,"threads_started":1}
I20260131 03:08:06.403415 25243 leader_election.cc:304] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ebbad9d6d24444cbc16f88fbde32079, 7abfc0422bf24c908a27d8e854be330d; no voters:
I20260131 03:08:06.403524 25176 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8e59dc4abdc402c93ea908d88361216" candidate_uuid: "2ebbad9d6d24444cbc16f88fbde32079" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe9b6ef2dce04a1982a805f3f3896a92" is_pre_election: true
I20260131 03:08:06.403584 25507 raft_consensus.cc:2804] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:08:06.403661 25507 raft_consensus.cc:493] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:08:06.403695 25507 raft_consensus.cc:3060] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:06.403692 25176 raft_consensus.cc:2468] T e8e59dc4abdc402c93ea908d88361216 P fe9b6ef2dce04a1982a805f3f3896a92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ebbad9d6d24444cbc16f88fbde32079 in term 0.
I20260131 03:08:06.404422 25507 raft_consensus.cc:515] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abfc0422bf24c908a27d8e854be330d" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 34555 } } peers { permanent_uuid: "2ebbad9d6d24444cbc16f88fbde32079" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 35437 } } peers { permanent_uuid: "fe9b6ef2dce04a1982a805f3f3896a92" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38737 } }
I20260131 03:08:06.404604 25507 leader_election.cc:290] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079 [CANDIDATE]: Term 1 election: Requested vote from peers 7abfc0422bf24c908a27d8e854be330d (127.19.84.3:34555), fe9b6ef2dce04a1982a805f3f3896a92 (127.19.84.1:38737)
I20260131 03:08:06.404752 25438 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8e59dc4abdc402c93ea908d88361216" candidate_uuid: "2ebbad9d6d24444cbc16f88fbde32079" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7abfc0422bf24c908a27d8e854be330d"
I20260131 03:08:06.404830 25438 raft_consensus.cc:3060] T e8e59dc4abdc402c93ea908d88361216 P 7abfc0422bf24c908a27d8e854be330d [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:06.404803 25176 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8e59dc4abdc402c93ea908d88361216" candidate_uuid: "2ebbad9d6d24444cbc16f88fbde32079" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe9b6ef2dce04a1982a805f3f3896a92"
I20260131 03:08:06.404902 25176 raft_consensus.cc:3060] T e8e59dc4abdc402c93ea908d88361216 P fe9b6ef2dce04a1982a805f3f3896a92 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:06.405849 25438 raft_consensus.cc:2468] T e8e59dc4abdc402c93ea908d88361216 P 7abfc0422bf24c908a27d8e854be330d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ebbad9d6d24444cbc16f88fbde32079 in term 1.
I20260131 03:08:06.405915 25176 raft_consensus.cc:2468] T e8e59dc4abdc402c93ea908d88361216 P fe9b6ef2dce04a1982a805f3f3896a92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ebbad9d6d24444cbc16f88fbde32079 in term 1.
I20260131 03:08:06.406075 25243 leader_election.cc:304] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ebbad9d6d24444cbc16f88fbde32079, 7abfc0422bf24c908a27d8e854be330d; no voters:
I20260131 03:08:06.406183 25507 raft_consensus.cc:2804] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:08:06.406317 25507 raft_consensus.cc:697] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079 [term 1 LEADER]: Becoming Leader. State: Replica: 2ebbad9d6d24444cbc16f88fbde32079, State: Running, Role: LEADER
I20260131 03:08:06.406441 25507 consensus_queue.cc:237] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abfc0422bf24c908a27d8e854be330d" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 34555 } } peers { permanent_uuid: "2ebbad9d6d24444cbc16f88fbde32079" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 35437 } } peers { permanent_uuid: "fe9b6ef2dce04a1982a805f3f3896a92" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38737 } }
I20260131 03:08:06.407233 25034 catalog_manager.cc:5654] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ebbad9d6d24444cbc16f88fbde32079 (127.19.84.2). New cstate: current_term: 1 leader_uuid: "2ebbad9d6d24444cbc16f88fbde32079" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7abfc0422bf24c908a27d8e854be330d" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 34555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ebbad9d6d24444cbc16f88fbde32079" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 35437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe9b6ef2dce04a1982a805f3f3896a92" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38737 } health_report { overall_health: UNKNOWN } } }
I20260131 03:08:06.437654 25496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:06.436276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42336 (local address 127.19.84.2:35437)
0131 03:08:06.436363 (+ 87us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:06.436366 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:06.436377 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:08:06.436498 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:06.436499 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:06.436528 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:06.436610 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:06.437395 (+ 785us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:06.437397 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:06.437398 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:08:06.437402 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:06.437427 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:08:06.437431 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:06.437498 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:06.437558 (+ 60us) server_negotiation.cc:300] Negotiation successful
0131 03:08:06.437574 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260131 03:08:06.438050 25485 tablet.cc:2378] T e8e59dc4abdc402c93ea908d88361216 P 7abfc0422bf24c908a27d8e854be330d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:08:06.439559 25438 raft_consensus.cc:1275] T e8e59dc4abdc402c93ea908d88361216 P 7abfc0422bf24c908a27d8e854be330d [term 1 FOLLOWER]: Refusing update from remote peer 2ebbad9d6d24444cbc16f88fbde32079: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:08:06.439560 25176 raft_consensus.cc:1275] T e8e59dc4abdc402c93ea908d88361216 P fe9b6ef2dce04a1982a805f3f3896a92 [term 1 FOLLOWER]: Refusing update from remote peer 2ebbad9d6d24444cbc16f88fbde32079: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:08:06.439888 25507 consensus_queue.cc:1048] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe9b6ef2dce04a1982a805f3f3896a92" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:08:06.440034 25509 consensus_queue.cc:1048] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7abfc0422bf24c908a27d8e854be330d" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 34555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:08:06.442790 25513 mvcc.cc:204] Tried to move back new op lower bound from 7249219118853472256 to 7249219118721351680. Current Snapshot: MvccSnapshot[applied={T|T < 7249219118853472256}]
I20260131 03:08:06.443800 25514 mvcc.cc:204] Tried to move back new op lower bound from 7249219118853472256 to 7249219118721351680. Current Snapshot: MvccSnapshot[applied={T|T < 7249219118853472256 or (T in {7249219118853472256})}]
I20260131 03:08:06.445072 25033 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:08:06.445163 25033 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:08:06.445998 25033 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e3bb3e394154486a9ec114340190f2cd: Sending DeleteTablet for 3 replicas of tablet e8e59dc4abdc402c93ea908d88361216
I20260131 03:08:06.446319 25156 tablet_service.cc:1552] Processing DeleteTablet for tablet e8e59dc4abdc402c93ea908d88361216 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:06 UTC) from {username='slave'} at 127.0.0.1:54586
I20260131 03:08:06.446374 25287 tablet_service.cc:1552] Processing DeleteTablet for tablet e8e59dc4abdc402c93ea908d88361216 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:06 UTC) from {username='slave'} at 127.0.0.1:42320
I20260131 03:08:06.446539 25418 tablet_service.cc:1552] Processing DeleteTablet for tablet e8e59dc4abdc402c93ea908d88361216 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:06 UTC) from {username='slave'} at 127.0.0.1:48874
I20260131 03:08:06.446682 25522 tablet_replica.cc:333] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079: stopping tablet replica
I20260131 03:08:06.446763 25521 tablet_replica.cc:333] T e8e59dc4abdc402c93ea908d88361216 P fe9b6ef2dce04a1982a805f3f3896a92: stopping tablet replica
I20260131 03:08:06.446858 25521 raft_consensus.cc:2243] T e8e59dc4abdc402c93ea908d88361216 P fe9b6ef2dce04a1982a805f3f3896a92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:08:06.446852 25522 raft_consensus.cc:2243] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:08:06.446904 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 25094
I20260131 03:08:06.446960 25521 raft_consensus.cc:2272] T e8e59dc4abdc402c93ea908d88361216 P fe9b6ef2dce04a1982a805f3f3896a92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:08:06.447104 25523 tablet_replica.cc:333] T e8e59dc4abdc402c93ea908d88361216 P 7abfc0422bf24c908a27d8e854be330d: stopping tablet replica
I20260131 03:08:06.447111 25522 raft_consensus.cc:2272] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:08:06.447177 25523 raft_consensus.cc:2243] T e8e59dc4abdc402c93ea908d88361216 P 7abfc0422bf24c908a27d8e854be330d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:08:06.447255 25523 raft_consensus.cc:2272] T e8e59dc4abdc402c93ea908d88361216 P 7abfc0422bf24c908a27d8e854be330d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:08:06.447798 25522 ts_tablet_manager.cc:1916] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:08:06.449611 25522 ts_tablet_manager.cc:1929] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:08:06.449713 25523 ts_tablet_manager.cc:1916] T e8e59dc4abdc402c93ea908d88361216 P 7abfc0422bf24c908a27d8e854be330d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:08:06.449733 25522 log.cc:1199] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-1/wal/wals/e8e59dc4abdc402c93ea908d88361216
I20260131 03:08:06.450106 25522 ts_tablet_manager.cc:1950] T e8e59dc4abdc402c93ea908d88361216 P 2ebbad9d6d24444cbc16f88fbde32079: Deleting consensus metadata
I20260131 03:08:06.450526 25021 catalog_manager.cc:4985] TS 2ebbad9d6d24444cbc16f88fbde32079 (127.19.84.2:35437): tablet e8e59dc4abdc402c93ea908d88361216 (table test-table [id=0f4c5a27b2fb411b95a0cfdbfe2b3173]) successfully deleted
I20260131 03:08:06.451341 25523 ts_tablet_manager.cc:1929] T e8e59dc4abdc402c93ea908d88361216 P 7abfc0422bf24c908a27d8e854be330d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:08:06.451438 25523 log.cc:1199] T e8e59dc4abdc402c93ea908d88361216 P 7abfc0422bf24c908a27d8e854be330d: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/e8e59dc4abdc402c93ea908d88361216
I20260131 03:08:06.451691 25523 ts_tablet_manager.cc:1950] T e8e59dc4abdc402c93ea908d88361216 P 7abfc0422bf24c908a27d8e854be330d: Deleting consensus metadata
I20260131 03:08:06.452135 25021 catalog_manager.cc:4985] TS 7abfc0422bf24c908a27d8e854be330d (127.19.84.3:34555): tablet e8e59dc4abdc402c93ea908d88361216 (table test-table [id=0f4c5a27b2fb411b95a0cfdbfe2b3173]) successfully deleted
W20260131 03:08:06.453423 25020 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.84.1:38737 (error 108)
I20260131 03:08:06.453958 25493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:06.453813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:38737 (local address 127.0.0.1:54594)
0131 03:08:06.453881 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:06.453922 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.1:38737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260131 03:08:06.454035 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 25225
W20260131 03:08:06.455111 25020 catalog_manager.cc:4712] TS fe9b6ef2dce04a1982a805f3f3896a92 (127.19.84.1:38737): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e8e59dc4abdc402c93ea908d88361216: Network error: Client connection negotiation failed: client connection to 127.19.84.1:38737: connect: Connection refused (error 111)
I20260131 03:08:06.458884 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 25356
I20260131 03:08:06.463624 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 25003
2026-01-31T03:08:06Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (816 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260131 03:08:06.483902 19792 mini_oidc.cc:150] Starting JWKS server
I20260131 03:08:06.484917 19792 webserver.cc:492] Webserver started at https://127.0.0.1:42801/ using document root <none> and password file <none>
I20260131 03:08:06.484987 19792 mini_oidc.cc:186] Starting OIDC Discovery server
I20260131 03:08:06.485121 19792 webserver.cc:492] Webserver started at http://127.0.0.1:40433/ using document root <none> and password file <none>
2026-01-31T03:08:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:08:06Z Disabled control of system clock
I20260131 03:08:06.492085 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:37109
--webserver_interface=127.19.84.62
--webserver_port=0
--builtin_ntp_servers=127.19.84.20:46161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:37109
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42801/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828849314684-19792-0/testchainca.cert with env {}
W20260131 03:08:06.570423 25530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:06.570609 25530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:06.570631 25530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:06.572160 25530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:08:06.572214 25530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:06.572227 25530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:06.572239 25530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:08:06.572252 25530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260131 03:08:06.572264 25530 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260131 03:08:06.572275 25530 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42801/jwks.json
I20260131 03:08:06.574067 25530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:46161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:37109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:37109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42801/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828849314684-19792-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:06.574328 25530 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:06.574542 25530 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260131 03:08:06.577917 25530 server_base.cc:1060] running on GCE node
W20260131 03:08:06.577888 25538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:06.577881 25536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:06.577893 25535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:06.578356 25530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:06.578583 25530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:06.579749 25530 hybrid_clock.cc:648] HybridClock initialized: now 1769828886579710 us; error 53 us; skew 500 ppm
I20260131 03:08:06.581005 25530 webserver.cc:492] Webserver started at http://127.19.84.62:37521/ using document root <none> and password file <none>
I20260131 03:08:06.581250 25530 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:06.581295 25530 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:06.581416 25530 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:06.582401 25530 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "ceec3f1673164a0295aa551cbaf9b8d8"
format_stamp: "Formatted at 2026-01-31 03:08:06 on dist-test-slave-5nnp"
server_key: "4a3d0eb78c555255e5e6da6d268a9d65"
server_key_iv: "6ed0028ebf7fc8fa61d5d29b76604176"
server_key_version: "encryptionkey@0"
I20260131 03:08:06.582736 25530 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "ceec3f1673164a0295aa551cbaf9b8d8"
format_stamp: "Formatted at 2026-01-31 03:08:06 on dist-test-slave-5nnp"
server_key: "4a3d0eb78c555255e5e6da6d268a9d65"
server_key_iv: "6ed0028ebf7fc8fa61d5d29b76604176"
server_key_version: "encryptionkey@0"
I20260131 03:08:06.584064 25530 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:08:06.584981 25544 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:06.585232 25530 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:06.585304 25530 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "ceec3f1673164a0295aa551cbaf9b8d8"
format_stamp: "Formatted at 2026-01-31 03:08:06 on dist-test-slave-5nnp"
server_key: "4a3d0eb78c555255e5e6da6d268a9d65"
server_key_iv: "6ed0028ebf7fc8fa61d5d29b76604176"
server_key_version: "encryptionkey@0"
I20260131 03:08:06.585386 25530 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:06.593677 25530 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:06.593982 25530 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:06.594112 25530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:06.598459 25530 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:37109
I20260131 03:08:06.598537 25596 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:37109 every 8 connection(s)
I20260131 03:08:06.598881 25530 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:08:06.599552 25597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:06.602138 25597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceec3f1673164a0295aa551cbaf9b8d8: Bootstrap starting.
I20260131 03:08:06.602751 25597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ceec3f1673164a0295aa551cbaf9b8d8: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:06.603025 25597 log.cc:826] T 00000000000000000000000000000000 P ceec3f1673164a0295aa551cbaf9b8d8: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:06.603945 25597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceec3f1673164a0295aa551cbaf9b8d8: No bootstrap required, opened a new log
I20260131 03:08:06.605180 25597 raft_consensus.cc:359] T 00000000000000000000000000000000 P ceec3f1673164a0295aa551cbaf9b8d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceec3f1673164a0295aa551cbaf9b8d8" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 37109 } }
I20260131 03:08:06.605309 25597 raft_consensus.cc:385] T 00000000000000000000000000000000 P ceec3f1673164a0295aa551cbaf9b8d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:06.605329 25597 raft_consensus.cc:740] T 00000000000000000000000000000000 P ceec3f1673164a0295aa551cbaf9b8d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceec3f1673164a0295aa551cbaf9b8d8, State: Initialized, Role: FOLLOWER
I20260131 03:08:06.605472 25597 consensus_queue.cc:260] T 00000000000000000000000000000000 P ceec3f1673164a0295aa551cbaf9b8d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceec3f1673164a0295aa551cbaf9b8d8" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 37109 } }
I20260131 03:08:06.605564 25597 raft_consensus.cc:399] T 00000000000000000000000000000000 P ceec3f1673164a0295aa551cbaf9b8d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:08:06.605616 25597 raft_consensus.cc:493] T 00000000000000000000000000000000 P ceec3f1673164a0295aa551cbaf9b8d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:08:06.605687 25597 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ceec3f1673164a0295aa551cbaf9b8d8 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:06.606004 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 25530
I20260131 03:08:06.606082 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:08:06.606282 19792 external_mini_cluster.cc:1468] Setting key 6017249da67f787fcfccf0470ca0b74f
I20260131 03:08:06.606411 25597 raft_consensus.cc:515] T 00000000000000000000000000000000 P ceec3f1673164a0295aa551cbaf9b8d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceec3f1673164a0295aa551cbaf9b8d8" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 37109 } }
I20260131 03:08:06.606539 25597 leader_election.cc:304] T 00000000000000000000000000000000 P ceec3f1673164a0295aa551cbaf9b8d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ceec3f1673164a0295aa551cbaf9b8d8; no voters:
I20260131 03:08:06.606746 25597 leader_election.cc:290] T 00000000000000000000000000000000 P ceec3f1673164a0295aa551cbaf9b8d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:08:06.606787 25600 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ceec3f1673164a0295aa551cbaf9b8d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:08:06.607071 25597 sys_catalog.cc:565] T 00000000000000000000000000000000 P ceec3f1673164a0295aa551cbaf9b8d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:08:06.607156 25600 raft_consensus.cc:697] T 00000000000000000000000000000000 P ceec3f1673164a0295aa551cbaf9b8d8 [term 1 LEADER]: Becoming Leader. State: Replica: ceec3f1673164a0295aa551cbaf9b8d8, State: Running, Role: LEADER
I20260131 03:08:06.607249 25600 consensus_queue.cc:237] T 00000000000000000000000000000000 P ceec3f1673164a0295aa551cbaf9b8d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceec3f1673164a0295aa551cbaf9b8d8" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 37109 } }
I20260131 03:08:06.607694 25602 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceec3f1673164a0295aa551cbaf9b8d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ceec3f1673164a0295aa551cbaf9b8d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceec3f1673164a0295aa551cbaf9b8d8" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 37109 } } }
I20260131 03:08:06.607880 25602 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceec3f1673164a0295aa551cbaf9b8d8 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:06.608053 25602 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceec3f1673164a0295aa551cbaf9b8d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ceec3f1673164a0295aa551cbaf9b8d8. Latest consensus state: current_term: 1 leader_uuid: "ceec3f1673164a0295aa551cbaf9b8d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceec3f1673164a0295aa551cbaf9b8d8" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 37109 } } }
I20260131 03:08:06.608158 25602 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceec3f1673164a0295aa551cbaf9b8d8 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:06.608424 25611 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:08:06.609336 25611 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:08:06.611605 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:06.606907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50806 (local address 127.19.84.62:37109)
0131 03:08:06.607287 (+ 380us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:06.607294 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:06.607315 (+ 21us) server_negotiation.cc:408] Connection header received
0131 03:08:06.607449 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:06.607455 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:06.607522 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:06.607660 (+ 138us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:06.608128 (+ 468us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:06.609364 (+ 1236us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:06.610135 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:06.610725 (+ 590us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:06.610849 (+ 124us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:06.610855 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:06.610856 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:08:06.611116 (+ 260us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:06.611142 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:08:06.611150 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:06.611258 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:06.611360 (+ 102us) server_negotiation.cc:300] Negotiation successful
0131 03:08:06.611452 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":62,"threads_started":1}
I20260131 03:08:06.611877 25611 catalog_manager.cc:1357] Generated new cluster ID: 4acbfc9c51bc43e1874ed2ff998e8dad
I20260131 03:08:06.611937 25611 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:08:06.612627 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:08:06.640574 25611 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:08:06.641377 25611 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:08:06.646469 25611 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ceec3f1673164a0295aa551cbaf9b8d8: Generated new TSK 0
I20260131 03:08:06.646764 25611 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:08:06.681279 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:06.660412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50810 (local address 127.19.84.62:37109)
0131 03:08:06.660462 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:06.660466 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:06.660763 (+ 297us) server_negotiation.cc:408] Connection header received
0131 03:08:06.660891 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:06.660892 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:06.660906 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:06.660993 (+ 87us) server_negotiation.cc:227] Negotiated authn=JWT
0131 03:08:06.661351 (+ 358us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:06.662155 (+ 804us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:06.663047 (+ 892us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:06.663284 (+ 237us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:06.663543 (+ 259us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0131 03:08:06.680955 (+ 17412us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
0131 03:08:06.681043 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:06.681139 (+ 96us) server_negotiation.cc:300] Negotiation successful
0131 03:08:06.681174 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26,"thread_start_us":110,"threads_started":1}
I20260131 03:08:06.689220 25603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:06.686233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50822 (local address 127.19.84.62:37109)
0131 03:08:06.686279 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:06.686282 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:06.686378 (+ 96us) server_negotiation.cc:408] Connection header received
0131 03:08:06.686446 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:06.686447 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:06.686456 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:06.686507 (+ 51us) server_negotiation.cc:227] Negotiated authn=JWT
0131 03:08:06.686895 (+ 388us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:06.687458 (+ 563us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:06.688401 (+ 943us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:06.688545 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:06.688602 (+ 57us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0131 03:08:06.688999 (+ 397us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0131 03:08:06.689173 (+ 174us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50822: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":24}
W20260131 03:08:06.689302 25603 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50822: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20260131 03:08:06.689481 25633 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:06.686139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:37109 (local address 127.0.0.1:50822)
0131 03:08:06.686297 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:06.686308 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:06.686382 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:06.686528 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:06.686529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:06.686531 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0131 03:08:06.686772 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:06.686774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:06.687553 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:06.687554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:06.688298 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:06.688301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:06.688392 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:06.688398 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0131 03:08:06.689152 (+ 754us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0131 03:08:06.689256 (+ 104us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.84.62:37109: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260131 03:08:21.696534 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:21.693509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41362 (local address 127.19.84.62:37109)
0131 03:08:21.693656 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:21.693659 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:21.693678 (+ 19us) server_negotiation.cc:408] Connection header received
0131 03:08:21.693711 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:21.693712 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:21.693725 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:21.693834 (+ 109us) server_negotiation.cc:227] Negotiated authn=JWT
0131 03:08:21.694191 (+ 357us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:21.694780 (+ 589us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:21.695809 (+ 1029us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:21.695994 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:21.696019 (+ 25us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0131 03:08:21.696284 (+ 265us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0131 03:08:21.696427 (+ 143us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41362: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
W20260131 03:08:21.696630 25639 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41362: JWT verification failed: token expired
W20260131 03:08:21.696727 25638 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:21.693325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:37109 (local address 127.0.0.1:41362)
0131 03:08:21.693537 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:21.693550 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:21.693630 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:21.693839 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:21.693840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:21.693843 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0131 03:08:21.694103 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:21.694105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:21.694894 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:21.694896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:21.695702 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:21.695706 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:21.695799 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:21.695806 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0131 03:08:21.696384 (+ 578us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0131 03:08:21.696546 (+ 162us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.84.62:37109: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":99,"threads_started":1}
W20260131 03:08:21.698539 25639 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:21.697962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41370 (local address 127.19.84.62:37109)
0131 03:08:21.698001 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:21.698003 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:21.698148 (+ 145us) server_negotiation.cc:408] Connection header received
0131 03:08:21.698198 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:21.698199 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:21.698213 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:21.698269 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:21.698492 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41370: BlockingRecv error: recv got EOF from 127.0.0.1:41370 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260131 03:08:21.701951 25649 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.19.84.62:37109
W20260131 03:08:21.702340 25639 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:21.701772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41378 (local address 127.19.84.62:37109)
0131 03:08:21.701822 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:21.701824 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:21.701950 (+ 126us) server_negotiation.cc:408] Connection header received
0131 03:08:21.702078 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:21.702079 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:21.702090 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:21.702139 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:21.702288 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41378: BlockingRecv error: recv got EOF from 127.0.0.1:41378 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260131 03:08:21.702661 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 25530
2026-01-31T03:08:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15257 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260131 03:08:21.741489 19792 mini_oidc.cc:150] Starting JWKS server
I20260131 03:08:21.742288 19792 webserver.cc:492] Webserver started at https://127.0.0.1:42265/ using document root <none> and password file <none>
I20260131 03:08:21.742359 19792 mini_oidc.cc:186] Starting OIDC Discovery server
I20260131 03:08:21.742511 19792 webserver.cc:492] Webserver started at http://127.0.0.1:39619/ using document root <none> and password file <none>
2026-01-31T03:08:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:08:21Z Disabled control of system clock
I20260131 03:08:21.749701 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:42377
--webserver_interface=127.19.84.62
--webserver_port=0
--builtin_ntp_servers=127.19.84.20:35571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:42377
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42265/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828849314684-19792-0/testchainca.cert with env {}
W20260131 03:08:21.830055 25655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:21.830291 25655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:21.830331 25655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:21.832015 25655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:08:21.832104 25655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:21.832125 25655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:21.832147 25655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:08:21.832173 25655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260131 03:08:21.832206 25655 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260131 03:08:21.832226 25655 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42265/jwks.json
I20260131 03:08:21.834393 25655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:35571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:42377
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:42377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42265/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828849314684-19792-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:21.834725 25655 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:21.834949 25655 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:21.838305 25660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:21.838382 25663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:21.838305 25661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:21.838500 25655 server_base.cc:1060] running on GCE node
I20260131 03:08:21.838763 25655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:21.839066 25655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:21.840253 25655 hybrid_clock.cc:648] HybridClock initialized: now 1769828901840198 us; error 64 us; skew 500 ppm
I20260131 03:08:21.841555 25655 webserver.cc:492] Webserver started at http://127.19.84.62:33973/ using document root <none> and password file <none>
I20260131 03:08:21.841866 25655 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:21.841940 25655 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:21.842144 25655 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:21.843360 25655 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "9291424336b54d1ba52817fb7774c441"
format_stamp: "Formatted at 2026-01-31 03:08:21 on dist-test-slave-5nnp"
server_key: "170106850f20d4dfdfc5f52d82fc9281"
server_key_iv: "12d2c011b2eddee2bb5ed9529d11bc7f"
server_key_version: "encryptionkey@0"
I20260131 03:08:21.843942 25655 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "9291424336b54d1ba52817fb7774c441"
format_stamp: "Formatted at 2026-01-31 03:08:21 on dist-test-slave-5nnp"
server_key: "170106850f20d4dfdfc5f52d82fc9281"
server_key_iv: "12d2c011b2eddee2bb5ed9529d11bc7f"
server_key_version: "encryptionkey@0"
I20260131 03:08:21.845276 25655 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260131 03:08:21.846195 25669 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:21.846531 25655 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:08:21.846621 25655 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "9291424336b54d1ba52817fb7774c441"
format_stamp: "Formatted at 2026-01-31 03:08:21 on dist-test-slave-5nnp"
server_key: "170106850f20d4dfdfc5f52d82fc9281"
server_key_iv: "12d2c011b2eddee2bb5ed9529d11bc7f"
server_key_version: "encryptionkey@0"
I20260131 03:08:21.846673 25655 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:21.854600 25655 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:21.854974 25655 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:21.855129 25655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:21.859025 25655 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:42377
I20260131 03:08:21.859086 25721 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:42377 every 8 connection(s)
I20260131 03:08:21.859499 25655 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:08:21.860203 25722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:21.862931 25722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9291424336b54d1ba52817fb7774c441: Bootstrap starting.
I20260131 03:08:21.863608 25722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9291424336b54d1ba52817fb7774c441: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:21.863713 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 25655
I20260131 03:08:21.863858 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:08:21.863956 25722 log.cc:826] T 00000000000000000000000000000000 P 9291424336b54d1ba52817fb7774c441: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:21.864058 19792 external_mini_cluster.cc:1468] Setting key 3d2b2caf250afef5f5efdf07a8d6b8ab
I20260131 03:08:21.865089 25722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9291424336b54d1ba52817fb7774c441: No bootstrap required, opened a new log
I20260131 03:08:21.866676 25722 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9291424336b54d1ba52817fb7774c441 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9291424336b54d1ba52817fb7774c441" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42377 } }
I20260131 03:08:21.866834 25722 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9291424336b54d1ba52817fb7774c441 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:21.866887 25722 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9291424336b54d1ba52817fb7774c441 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9291424336b54d1ba52817fb7774c441, State: Initialized, Role: FOLLOWER
I20260131 03:08:21.867002 25722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9291424336b54d1ba52817fb7774c441 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9291424336b54d1ba52817fb7774c441" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42377 } }
I20260131 03:08:21.867081 25722 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9291424336b54d1ba52817fb7774c441 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:08:21.867113 25722 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9291424336b54d1ba52817fb7774c441 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:08:21.867153 25722 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9291424336b54d1ba52817fb7774c441 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:21.868014 25722 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9291424336b54d1ba52817fb7774c441 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9291424336b54d1ba52817fb7774c441" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42377 } }
I20260131 03:08:21.868153 25722 leader_election.cc:304] T 00000000000000000000000000000000 P 9291424336b54d1ba52817fb7774c441 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9291424336b54d1ba52817fb7774c441; no voters:
I20260131 03:08:21.868367 25722 leader_election.cc:290] T 00000000000000000000000000000000 P 9291424336b54d1ba52817fb7774c441 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:08:21.868438 25727 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9291424336b54d1ba52817fb7774c441 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:08:21.868574 25727 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9291424336b54d1ba52817fb7774c441 [term 1 LEADER]: Becoming Leader. State: Replica: 9291424336b54d1ba52817fb7774c441, State: Running, Role: LEADER
I20260131 03:08:21.868660 25722 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9291424336b54d1ba52817fb7774c441 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:08:21.868666 25727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9291424336b54d1ba52817fb7774c441 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9291424336b54d1ba52817fb7774c441" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42377 } }
I20260131 03:08:21.869011 25728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9291424336b54d1ba52817fb7774c441 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9291424336b54d1ba52817fb7774c441" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9291424336b54d1ba52817fb7774c441" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42377 } } }
I20260131 03:08:21.869114 25728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9291424336b54d1ba52817fb7774c441 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:21.869020 25729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9291424336b54d1ba52817fb7774c441 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9291424336b54d1ba52817fb7774c441. Latest consensus state: current_term: 1 leader_uuid: "9291424336b54d1ba52817fb7774c441" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9291424336b54d1ba52817fb7774c441" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42377 } } }
I20260131 03:08:21.869261 25729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9291424336b54d1ba52817fb7774c441 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:21.869315 25731 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:08:21.869534 25726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:21.864781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.19.84.62:42377)
0131 03:08:21.865092 (+ 311us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:21.865099 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:21.865120 (+ 21us) server_negotiation.cc:408] Connection header received
0131 03:08:21.865527 (+ 407us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:21.865533 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:21.865584 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:21.865699 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:21.866177 (+ 478us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:21.866985 (+ 808us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:21.867807 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:21.867990 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:21.868576 (+ 586us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:21.868585 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:21.868585 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:08:21.868820 (+ 235us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:21.868855 (+ 35us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:08:21.868865 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:21.868947 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:21.869092 (+ 145us) server_negotiation.cc:300] Negotiation successful
0131 03:08:21.869149 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":186,"threads_started":1}
I20260131 03:08:21.870039 25731 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:08:21.870604 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:08:21.876024 25731 catalog_manager.cc:1357] Generated new cluster ID: 231c760819be4fc393249583424e3123
I20260131 03:08:21.876113 25731 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:08:21.894601 25731 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:08:21.895365 25731 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:08:21.914739 25731 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 9291424336b54d1ba52817fb7774c441: Generated new TSK 0
I20260131 03:08:21.914988 25731 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260131 03:08:21.936714 25752 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260131 03:08:21.936836 25752 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260131 03:08:21.937124 25726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:21.921823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33688 (local address 127.19.84.62:42377)
0131 03:08:21.921903 (+ 80us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:21.921906 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:21.922081 (+ 175us) server_negotiation.cc:408] Connection header received
0131 03:08:21.922146 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:21.922147 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:21.922162 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:21.922255 (+ 93us) server_negotiation.cc:227] Negotiated authn=JWT
0131 03:08:21.922549 (+ 294us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:21.923363 (+ 814us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:21.924103 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:21.924294 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:21.924353 (+ 59us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0131 03:08:21.936896 (+ 12543us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0131 03:08:21.937046 (+ 150us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33688: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":53}
W20260131 03:08:21.937211 25726 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33688: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260131 03:08:21.937229 25751 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:21.921684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:42377 (local address 127.0.0.1:33688)
0131 03:08:21.921888 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:21.921898 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:21.921947 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:21.922267 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:21.922267 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:21.922269 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0131 03:08:21.922453 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:21.922455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:21.923555 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:21.923557 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:21.924005 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:21.924007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:21.924079 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:21.924084 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0131 03:08:21.937004 (+ 12920us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0131 03:08:21.937117 (+ 113us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.84.62:42377: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":95,"threads_started":1}
I20260131 03:08:21.937676 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 25655
2026-01-31T03:08:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (233 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260131 03:08:21.975509 19792 mini_oidc.cc:150] Starting JWKS server
I20260131 03:08:21.976248 19792 webserver.cc:492] Webserver started at https://127.0.0.1:40859/ using document root <none> and password file <none>
I20260131 03:08:21.976289 19792 mini_oidc.cc:186] Starting OIDC Discovery server
I20260131 03:08:21.976408 19792 webserver.cc:492] Webserver started at http://127.0.0.1:44613/ using document root <none> and password file <none>
2026-01-31T03:08:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:08:21Z Disabled control of system clock
I20260131 03:08:21.983950 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:40475
--webserver_interface=127.19.84.62
--webserver_port=0
--builtin_ntp_servers=127.19.84.20:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:40475
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40859/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260131 03:08:22.069615 25758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:22.069803 25758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:22.069826 25758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:22.071290 25758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:08:22.071337 25758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:22.071352 25758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:22.071364 25758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:08:22.071431 25758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260131 03:08:22.071453 25758 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260131 03:08:22.071470 25758 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40859/jwks.json
I20260131 03:08:22.073035 25758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:40475
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:40475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40859/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:22.073292 25758 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:22.073530 25758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:22.076290 25766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:22.076342 25763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:22.076290 25764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:22.076377 25758 server_base.cc:1060] running on GCE node
I20260131 03:08:22.076789 25758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:22.077049 25758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:22.078220 25758 hybrid_clock.cc:648] HybridClock initialized: now 1769828902078204 us; error 47 us; skew 500 ppm
I20260131 03:08:22.079516 25758 webserver.cc:492] Webserver started at http://127.19.84.62:46493/ using document root <none> and password file <none>
I20260131 03:08:22.079746 25758 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:22.079809 25758 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:22.079942 25758 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:22.080933 25758 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "8651dc6505a2410f8ca5abc10ad75423"
format_stamp: "Formatted at 2026-01-31 03:08:22 on dist-test-slave-5nnp"
server_key: "78047804fac257470d9584b3384a518f"
server_key_iv: "fda8c0e28372b90abc5820041f5ae103"
server_key_version: "encryptionkey@0"
I20260131 03:08:22.081259 25758 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "8651dc6505a2410f8ca5abc10ad75423"
format_stamp: "Formatted at 2026-01-31 03:08:22 on dist-test-slave-5nnp"
server_key: "78047804fac257470d9584b3384a518f"
server_key_iv: "fda8c0e28372b90abc5820041f5ae103"
server_key_version: "encryptionkey@0"
I20260131 03:08:22.082504 25758 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260131 03:08:22.083245 25772 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:22.083509 25758 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:08:22.083602 25758 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "8651dc6505a2410f8ca5abc10ad75423"
format_stamp: "Formatted at 2026-01-31 03:08:22 on dist-test-slave-5nnp"
server_key: "78047804fac257470d9584b3384a518f"
server_key_iv: "fda8c0e28372b90abc5820041f5ae103"
server_key_version: "encryptionkey@0"
I20260131 03:08:22.083684 25758 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:22.095321 25758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:22.095697 25758 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:22.095800 25758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:22.099357 25758 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:40475
I20260131 03:08:22.099444 25824 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:40475 every 8 connection(s)
I20260131 03:08:22.099839 25758 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:08:22.100467 25825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:22.102845 25825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8651dc6505a2410f8ca5abc10ad75423: Bootstrap starting.
I20260131 03:08:22.103569 25825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8651dc6505a2410f8ca5abc10ad75423: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:22.103826 25825 log.cc:826] T 00000000000000000000000000000000 P 8651dc6505a2410f8ca5abc10ad75423: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:22.104584 25825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8651dc6505a2410f8ca5abc10ad75423: No bootstrap required, opened a new log
I20260131 03:08:22.105810 25825 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8651dc6505a2410f8ca5abc10ad75423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8651dc6505a2410f8ca5abc10ad75423" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 40475 } }
I20260131 03:08:22.105962 25825 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8651dc6505a2410f8ca5abc10ad75423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:22.106004 25825 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8651dc6505a2410f8ca5abc10ad75423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8651dc6505a2410f8ca5abc10ad75423, State: Initialized, Role: FOLLOWER
I20260131 03:08:22.106155 25825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8651dc6505a2410f8ca5abc10ad75423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8651dc6505a2410f8ca5abc10ad75423" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 40475 } }
I20260131 03:08:22.106235 25825 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8651dc6505a2410f8ca5abc10ad75423 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:08:22.106279 25825 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8651dc6505a2410f8ca5abc10ad75423 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:08:22.106334 25825 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8651dc6505a2410f8ca5abc10ad75423 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:22.106897 25825 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8651dc6505a2410f8ca5abc10ad75423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8651dc6505a2410f8ca5abc10ad75423" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 40475 } }
I20260131 03:08:22.107028 25825 leader_election.cc:304] T 00000000000000000000000000000000 P 8651dc6505a2410f8ca5abc10ad75423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8651dc6505a2410f8ca5abc10ad75423; no voters:
I20260131 03:08:22.107185 25825 leader_election.cc:290] T 00000000000000000000000000000000 P 8651dc6505a2410f8ca5abc10ad75423 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:08:22.107246 25828 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8651dc6505a2410f8ca5abc10ad75423 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:08:22.107462 25828 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8651dc6505a2410f8ca5abc10ad75423 [term 1 LEADER]: Becoming Leader. State: Replica: 8651dc6505a2410f8ca5abc10ad75423, State: Running, Role: LEADER
I20260131 03:08:22.107463 25825 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8651dc6505a2410f8ca5abc10ad75423 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:08:22.107568 25828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8651dc6505a2410f8ca5abc10ad75423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8651dc6505a2410f8ca5abc10ad75423" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 40475 } }
I20260131 03:08:22.107964 25829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8651dc6505a2410f8ca5abc10ad75423 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8651dc6505a2410f8ca5abc10ad75423" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8651dc6505a2410f8ca5abc10ad75423" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 40475 } } }
I20260131 03:08:22.108090 25829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8651dc6505a2410f8ca5abc10ad75423 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:22.107965 25830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8651dc6505a2410f8ca5abc10ad75423 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8651dc6505a2410f8ca5abc10ad75423. Latest consensus state: current_term: 1 leader_uuid: "8651dc6505a2410f8ca5abc10ad75423" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8651dc6505a2410f8ca5abc10ad75423" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 40475 } } }
I20260131 03:08:22.108219 25830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8651dc6505a2410f8ca5abc10ad75423 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:22.108338 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 25758
I20260131 03:08:22.108444 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:08:22.108529 25836 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:08:22.108603 19792 external_mini_cluster.cc:1468] Setting key 522e522ed0e87d6d27bfae9912607ba5
I20260131 03:08:22.109081 25836 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:08:22.110670 25836 catalog_manager.cc:1357] Generated new cluster ID: dbf8aa78fed64445bc6503475fb6aee3
I20260131 03:08:22.110724 25836 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:08:22.113437 25845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:22.109048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43666 (local address 127.19.84.62:40475)
0131 03:08:22.109239 (+ 191us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:22.109247 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:22.109725 (+ 478us) server_negotiation.cc:408] Connection header received
0131 03:08:22.109864 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:22.109868 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:22.109922 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:22.110051 (+ 129us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:22.110527 (+ 476us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:22.111478 (+ 951us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:22.112088 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:22.112677 (+ 589us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:22.112826 (+ 149us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:22.112845 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:22.112845 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:08:22.113022 (+ 177us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:22.113045 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:08:22.113051 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:22.113139 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:22.113220 (+ 81us) server_negotiation.cc:300] Negotiation successful
0131 03:08:22.113281 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
I20260131 03:08:22.114303 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:08:22.135958 25836 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:08:22.136574 25836 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:08:22.145656 25836 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 8651dc6505a2410f8ca5abc10ad75423: Generated new TSK 0
I20260131 03:08:22.145895 25836 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260131 03:08:22.177289 25855 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260131 03:08:22.177371 25855 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260131 03:08:22.178421 25845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:22.157668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43682 (local address 127.19.84.62:40475)
0131 03:08:22.157735 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:22.157738 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:22.157856 (+ 118us) server_negotiation.cc:408] Connection header received
0131 03:08:22.157946 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:22.157947 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:22.157959 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:22.158012 (+ 53us) server_negotiation.cc:227] Negotiated authn=JWT
0131 03:08:22.158394 (+ 382us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:22.158928 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:22.159826 (+ 898us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:22.159977 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:22.160027 (+ 50us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0131 03:08:22.178189 (+ 18162us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40859/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0131 03:08:22.178367 (+ 178us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43682: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40859/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":38}
W20260131 03:08:22.178494 25845 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43682: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40859/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260131 03:08:22.178645 25854 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:22.157600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:40475 (local address 127.0.0.1:43682)
0131 03:08:22.157777 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:22.157788 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:22.157863 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:22.158036 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:22.158037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:22.158039 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0131 03:08:22.158293 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:22.158294 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:22.159027 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:22.159028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:22.159729 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:22.159732 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:22.159830 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:22.159835 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0131 03:08:22.178311 (+ 18476us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40859/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0131 03:08:22.178454 (+ 143us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.84.62:40475: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40859/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
I20260131 03:08:22.178972 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 25758
2026-01-31T03:08:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (241 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-01-31T03:08:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:08:22Z Disabled control of system clock
I20260131 03:08:22.224294 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:44809
--webserver_interface=127.19.84.62
--webserver_port=0
--builtin_ntp_servers=127.19.84.20:43585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:44809
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828849314684-19792-0/testchainca.cert with env {}
W20260131 03:08:22.302572 25859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:22.302753 25859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:22.302775 25859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:22.304416 25859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:08:22.304463 25859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:22.304481 25859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:22.304493 25859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:08:22.304543 25859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260131 03:08:22.304558 25859 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260131 03:08:22.304582 25859 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260131 03:08:22.306223 25859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:43585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:44809
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:44809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828849314684-19792-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:22.306468 25859 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:22.306653 25859 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:22.309422 25864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:22.309499 25867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:22.309407 25865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:22.309506 25859 server_base.cc:1060] running on GCE node
I20260131 03:08:22.309895 25859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:22.310186 25859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:22.311338 25859 hybrid_clock.cc:648] HybridClock initialized: now 1769828902311329 us; error 29 us; skew 500 ppm
I20260131 03:08:22.312551 25859 webserver.cc:492] Webserver started at http://127.19.84.62:46511/ using document root <none> and password file <none>
I20260131 03:08:22.312772 25859 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:22.312845 25859 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:22.312985 25859 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:22.313934 25859 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "34144c060fae4926b5616db2ea40e14c"
format_stamp: "Formatted at 2026-01-31 03:08:22 on dist-test-slave-5nnp"
server_key: "5957447d4e859a967ff406af6af382eb"
server_key_iv: "cf9d33393b95d95c1b01e2caab08ea5a"
server_key_version: "encryptionkey@0"
I20260131 03:08:22.314258 25859 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "34144c060fae4926b5616db2ea40e14c"
format_stamp: "Formatted at 2026-01-31 03:08:22 on dist-test-slave-5nnp"
server_key: "5957447d4e859a967ff406af6af382eb"
server_key_iv: "cf9d33393b95d95c1b01e2caab08ea5a"
server_key_version: "encryptionkey@0"
I20260131 03:08:22.315510 25859 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260131 03:08:22.316356 25873 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:22.316613 25859 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260131 03:08:22.316711 25859 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "34144c060fae4926b5616db2ea40e14c"
format_stamp: "Formatted at 2026-01-31 03:08:22 on dist-test-slave-5nnp"
server_key: "5957447d4e859a967ff406af6af382eb"
server_key_iv: "cf9d33393b95d95c1b01e2caab08ea5a"
server_key_version: "encryptionkey@0"
I20260131 03:08:22.316813 25859 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:22.328469 25859 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:22.328794 25859 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:22.328935 25859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:22.333110 25859 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:44809
I20260131 03:08:22.333174 25925 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:44809 every 8 connection(s)
I20260131 03:08:22.333505 25859 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:08:22.334316 25926 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:22.336781 25926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34144c060fae4926b5616db2ea40e14c: Bootstrap starting.
I20260131 03:08:22.337383 25926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34144c060fae4926b5616db2ea40e14c: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:22.337643 25926 log.cc:826] T 00000000000000000000000000000000 P 34144c060fae4926b5616db2ea40e14c: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:22.338326 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 25859
I20260131 03:08:22.338433 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:08:22.338431 25926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34144c060fae4926b5616db2ea40e14c: No bootstrap required, opened a new log
I20260131 03:08:22.338608 19792 external_mini_cluster.cc:1468] Setting key 737d6e5764afb0bc55de2c8540d9a8c1
I20260131 03:08:22.340068 25926 raft_consensus.cc:359] T 00000000000000000000000000000000 P 34144c060fae4926b5616db2ea40e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34144c060fae4926b5616db2ea40e14c" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44809 } }
I20260131 03:08:22.340215 25926 raft_consensus.cc:385] T 00000000000000000000000000000000 P 34144c060fae4926b5616db2ea40e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:22.340257 25926 raft_consensus.cc:740] T 00000000000000000000000000000000 P 34144c060fae4926b5616db2ea40e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34144c060fae4926b5616db2ea40e14c, State: Initialized, Role: FOLLOWER
I20260131 03:08:22.340363 25926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34144c060fae4926b5616db2ea40e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34144c060fae4926b5616db2ea40e14c" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44809 } }
I20260131 03:08:22.340433 25926 raft_consensus.cc:399] T 00000000000000000000000000000000 P 34144c060fae4926b5616db2ea40e14c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:08:22.340476 25926 raft_consensus.cc:493] T 00000000000000000000000000000000 P 34144c060fae4926b5616db2ea40e14c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:08:22.340529 25926 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 34144c060fae4926b5616db2ea40e14c [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:22.341167 25926 raft_consensus.cc:515] T 00000000000000000000000000000000 P 34144c060fae4926b5616db2ea40e14c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34144c060fae4926b5616db2ea40e14c" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44809 } }
I20260131 03:08:22.341298 25926 leader_election.cc:304] T 00000000000000000000000000000000 P 34144c060fae4926b5616db2ea40e14c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34144c060fae4926b5616db2ea40e14c; no voters:
I20260131 03:08:22.341480 25926 leader_election.cc:290] T 00000000000000000000000000000000 P 34144c060fae4926b5616db2ea40e14c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:08:22.341512 25931 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 34144c060fae4926b5616db2ea40e14c [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:08:22.341627 25931 raft_consensus.cc:697] T 00000000000000000000000000000000 P 34144c060fae4926b5616db2ea40e14c [term 1 LEADER]: Becoming Leader. State: Replica: 34144c060fae4926b5616db2ea40e14c, State: Running, Role: LEADER
I20260131 03:08:22.341703 25931 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34144c060fae4926b5616db2ea40e14c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34144c060fae4926b5616db2ea40e14c" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44809 } }
I20260131 03:08:22.341729 25926 sys_catalog.cc:565] T 00000000000000000000000000000000 P 34144c060fae4926b5616db2ea40e14c [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:08:22.342039 25931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34144c060fae4926b5616db2ea40e14c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34144c060fae4926b5616db2ea40e14c. Latest consensus state: current_term: 1 leader_uuid: "34144c060fae4926b5616db2ea40e14c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34144c060fae4926b5616db2ea40e14c" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44809 } } }
I20260131 03:08:22.342104 25931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34144c060fae4926b5616db2ea40e14c [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:22.342083 25932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34144c060fae4926b5616db2ea40e14c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34144c060fae4926b5616db2ea40e14c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34144c060fae4926b5616db2ea40e14c" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44809 } } }
I20260131 03:08:22.342316 25932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34144c060fae4926b5616db2ea40e14c [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:22.343492 25930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:22.339122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57614 (local address 127.19.84.62:44809)
0131 03:08:22.339352 (+ 230us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:22.339363 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:22.339424 (+ 61us) server_negotiation.cc:408] Connection header received
0131 03:08:22.339832 (+ 408us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:22.339839 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:22.339889 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:22.340028 (+ 139us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:22.340421 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:22.341252 (+ 831us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:22.342289 (+ 1037us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:22.342584 (+ 295us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:22.342701 (+ 117us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:22.342708 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:22.342708 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:08:22.342891 (+ 183us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:22.342913 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:08:22.342920 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:22.343006 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:22.343068 (+ 62us) server_negotiation.cc:300] Negotiation successful
0131 03:08:22.343124 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260131 03:08:22.343694 25935 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:08:22.344498 25935 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:08:22.344650 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:08:22.345999 25935 catalog_manager.cc:1357] Generated new cluster ID: 042ebedbd321428fa7f244a195d2a06d
I20260131 03:08:22.346053 25935 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:08:22.359871 25935 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:08:22.360618 25935 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:08:22.373618 25935 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 34144c060fae4926b5616db2ea40e14c: Generated new TSK 0
I20260131 03:08:22.373850 25935 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:08:22.447937 25930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:22.381621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57624 (local address 127.19.84.62:44809)
0131 03:08:22.381671 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:22.381673 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:22.381791 (+ 118us) server_negotiation.cc:408] Connection header received
0131 03:08:22.381866 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:22.381867 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:22.381878 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:22.381933 (+ 55us) server_negotiation.cc:227] Negotiated authn=JWT
0131 03:08:22.382288 (+ 355us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:22.382795 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:22.383646 (+ 851us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:22.383766 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:22.383804 (+ 38us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0131 03:08:22.447677 (+ 63873us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0131 03:08:22.447854 (+ 177us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57624: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":23}
W20260131 03:08:22.448026 25930 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57624: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
W20260131 03:08:22.448082 25955 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:22.381532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44809 (local address 127.0.0.1:57624)
0131 03:08:22.381704 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:22.381714 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:22.381792 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:22.381968 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:22.381969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:22.381971 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0131 03:08:22.382220 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:22.382221 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:22.382889 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:22.382890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:22.383571 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:22.383574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:22.383662 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:22.383667 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0131 03:08:22.447820 (+ 64153us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0131 03:08:22.447974 (+ 154us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.84.62:44809: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":80,"threads_started":1}
I20260131 03:08:22.448486 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 25859
2026-01-31T03:08:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (249 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (85 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:08:22 dist-test-slave-5nnp krb5kdc[25967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:08:22 dist-test-slave-5nnp krb5kdc[25967](info): set up 2 sockets
Jan 31 03:08:22 dist-test-slave-5nnp krb5kdc[25967](info): commencing operation
krb5kdc: starting...
W20260131 03:08:24.616111 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.976s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:08:24 dist-test-slave-5nnp krb5kdc[25967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:08:24Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:08:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:08:24.751909 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:33835
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:42613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:33835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:24.830142 25983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:24.830358 25983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:24.830396 25983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:24.831876 25983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:08:24.831962 25983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:24.831985 25983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:24.832005 25983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:08:24.832032 25983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:08:24.833701 25983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:42613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:33835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:33835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:24.833994 25983 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:24.834195 25983 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:24.837113 25989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:24.837101 25983 server_base.cc:1060] running on GCE node
W20260131 03:08:24.837103 25988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:24.837229 25991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:24.837536 25983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:24.837813 25983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:24.838984 25983 hybrid_clock.cc:648] HybridClock initialized: now 1769828904838971 us; error 42 us; skew 500 ppm
Jan 31 03:08:24 dist-test-slave-5nnp krb5kdc[25967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828904, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:24.841609 25983 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:08:24.842245 25983 webserver.cc:492] Webserver started at http://127.19.84.62:36105/ using document root <none> and password file <none>
I20260131 03:08:24.842471 25983 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:24.842528 25983 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:24.842669 25983 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:24.843789 25983 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "63ea8b378aba4320b3e6262518f10006"
format_stamp: "Formatted at 2026-01-31 03:08:24 on dist-test-slave-5nnp"
server_key: "5d2685a7239a5f4302ad09f983137e20"
server_key_iv: "adcc0e6422088357c647495209cc2d1f"
server_key_version: "encryptionkey@0"
I20260131 03:08:24.844113 25983 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "63ea8b378aba4320b3e6262518f10006"
format_stamp: "Formatted at 2026-01-31 03:08:24 on dist-test-slave-5nnp"
server_key: "5d2685a7239a5f4302ad09f983137e20"
server_key_iv: "adcc0e6422088357c647495209cc2d1f"
server_key_version: "encryptionkey@0"
I20260131 03:08:24.845459 25983 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260131 03:08:24.846336 25998 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:24.846611 25983 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:08:24.846709 25983 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "63ea8b378aba4320b3e6262518f10006"
format_stamp: "Formatted at 2026-01-31 03:08:24 on dist-test-slave-5nnp"
server_key: "5d2685a7239a5f4302ad09f983137e20"
server_key_iv: "adcc0e6422088357c647495209cc2d1f"
server_key_version: "encryptionkey@0"
I20260131 03:08:24.846796 25983 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:24.868969 25983 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:24.871078 25983 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:24.871233 25983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:24.875286 25983 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:33835
I20260131 03:08:24.875339 26050 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:33835 every 8 connection(s)
I20260131 03:08:24.875734 25983 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:08:24.876386 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 25983
I20260131 03:08:24.876523 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:08:24.876505 26051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:24.876717 19792 external_mini_cluster.cc:1468] Setting key 770caf8d09b07569288723d3a939540a
I20260131 03:08:24.879992 26051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63ea8b378aba4320b3e6262518f10006: Bootstrap starting.
I20260131 03:08:24.880672 26051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63ea8b378aba4320b3e6262518f10006: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:24.880942 26051 log.cc:826] T 00000000000000000000000000000000 P 63ea8b378aba4320b3e6262518f10006: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:24.881695 26051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63ea8b378aba4320b3e6262518f10006: No bootstrap required, opened a new log
I20260131 03:08:24.882912 26051 raft_consensus.cc:359] T 00000000000000000000000000000000 P 63ea8b378aba4320b3e6262518f10006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ea8b378aba4320b3e6262518f10006" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33835 } }
I20260131 03:08:24.883064 26051 raft_consensus.cc:385] T 00000000000000000000000000000000 P 63ea8b378aba4320b3e6262518f10006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:24.883093 26051 raft_consensus.cc:740] T 00000000000000000000000000000000 P 63ea8b378aba4320b3e6262518f10006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63ea8b378aba4320b3e6262518f10006, State: Initialized, Role: FOLLOWER
Jan 31 03:08:24 dist-test-slave-5nnp krb5kdc[25967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:08:24.883189 26051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63ea8b378aba4320b3e6262518f10006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ea8b378aba4320b3e6262518f10006" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33835 } }
I20260131 03:08:24.883257 26051 raft_consensus.cc:399] T 00000000000000000000000000000000 P 63ea8b378aba4320b3e6262518f10006 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:08:24.883301 26051 raft_consensus.cc:493] T 00000000000000000000000000000000 P 63ea8b378aba4320b3e6262518f10006 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:08:24.883373 26051 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 63ea8b378aba4320b3e6262518f10006 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:24.884119 26051 raft_consensus.cc:515] T 00000000000000000000000000000000 P 63ea8b378aba4320b3e6262518f10006 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ea8b378aba4320b3e6262518f10006" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33835 } }
I20260131 03:08:24.884243 26051 leader_election.cc:304] T 00000000000000000000000000000000 P 63ea8b378aba4320b3e6262518f10006 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63ea8b378aba4320b3e6262518f10006; no voters:
I20260131 03:08:24.884423 26051 leader_election.cc:290] T 00000000000000000000000000000000 P 63ea8b378aba4320b3e6262518f10006 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:08:24.884567 26056 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 63ea8b378aba4320b3e6262518f10006 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:08:24.884684 26051 sys_catalog.cc:565] T 00000000000000000000000000000000 P 63ea8b378aba4320b3e6262518f10006 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:08:24.884725 26056 raft_consensus.cc:697] T 00000000000000000000000000000000 P 63ea8b378aba4320b3e6262518f10006 [term 1 LEADER]: Becoming Leader. State: Replica: 63ea8b378aba4320b3e6262518f10006, State: Running, Role: LEADER
I20260131 03:08:24.884832 26056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63ea8b378aba4320b3e6262518f10006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ea8b378aba4320b3e6262518f10006" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33835 } }
I20260131 03:08:24.885125 26056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63ea8b378aba4320b3e6262518f10006 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63ea8b378aba4320b3e6262518f10006. Latest consensus state: current_term: 1 leader_uuid: "63ea8b378aba4320b3e6262518f10006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ea8b378aba4320b3e6262518f10006" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33835 } } }
I20260131 03:08:24.885213 26056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63ea8b378aba4320b3e6262518f10006 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:24.885445 26062 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:08:24.885987 26062 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:08:24.887136 26057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63ea8b378aba4320b3e6262518f10006 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63ea8b378aba4320b3e6262518f10006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63ea8b378aba4320b3e6262518f10006" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33835 } } }
I20260131 03:08:24.887216 26057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63ea8b378aba4320b3e6262518f10006 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:24.887753 26062 catalog_manager.cc:1357] Generated new cluster ID: 1d18f33c58fa4dea83c768ff491b1487
I20260131 03:08:24.887809 26062 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:08:24.888407 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:24.877259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46536 (local address 127.19.84.62:33835)
0131 03:08:24.877416 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:24.877429 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:24.877457 (+ 28us) server_negotiation.cc:408] Connection header received
0131 03:08:24.878430 (+ 973us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:24.878439 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:24.878500 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:24.878646 (+ 146us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:24.879341 (+ 695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:24.880517 (+ 1176us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:24.881173 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:24.881393 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:24.883742 (+ 2349us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:24.883751 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:24.883752 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:24.883774 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:24.886803 (+ 3029us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:24.887332 (+ 529us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:24.887335 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:24.887336 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:24.887431 (+ 95us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:24.887628 (+ 197us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:24.887629 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:24.887629 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:24.887886 (+ 257us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:24.888058 (+ 172us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:24.888132 (+ 74us) server_negotiation.cc:300] Negotiation successful
0131 03:08:24.888248 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260131 03:08:24.901001 26062 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:08:24.901800 26062 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:08:24.908087 26062 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 63ea8b378aba4320b3e6262518f10006: Generated new TSK 0
I20260131 03:08:24.908329 26062 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:08:24.966056 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:33835
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:42613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:25.052358 26079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:25.052547 26079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:25.052573 26079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:25.054095 26079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:25.054154 26079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:25.054183 26079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:08:25.056005 26079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:42613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:33835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:25.056237 26079 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:25.056473 26079 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:25.059581 26084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:25.059619 26079 server_base.cc:1060] running on GCE node
W20260131 03:08:25.059619 26085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:25.060156 26087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:25.060371 26079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:25.060570 26079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:25.061729 26079 hybrid_clock.cc:648] HybridClock initialized: now 1769828905061713 us; error 28 us; skew 500 ppm
Jan 31 03:08:25 dist-test-slave-5nnp krb5kdc[25967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828905, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:25.064339 26079 init.cc:377] Logged in from keytab as kudu/127.19.84.1@KRBTEST.COM (short username kudu)
I20260131 03:08:25.064896 26079 webserver.cc:492] Webserver started at http://127.19.84.1:37457/ using document root <none> and password file <none>
I20260131 03:08:25.065053 26079 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:25.065086 26079 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:25.065166 26079 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:25.066092 26079 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "5f50a746500b4753b54d5cf1dfd9cfa1"
format_stamp: "Formatted at 2026-01-31 03:08:25 on dist-test-slave-5nnp"
server_key: "01c2911e47233bede52b8b4ecfc71e07"
server_key_iv: "923867bdb6dbc30668b7fa7468bf93fb"
server_key_version: "encryptionkey@0"
I20260131 03:08:25.066367 26079 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "5f50a746500b4753b54d5cf1dfd9cfa1"
format_stamp: "Formatted at 2026-01-31 03:08:25 on dist-test-slave-5nnp"
server_key: "01c2911e47233bede52b8b4ecfc71e07"
server_key_iv: "923867bdb6dbc30668b7fa7468bf93fb"
server_key_version: "encryptionkey@0"
I20260131 03:08:25.067677 26079 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:08:25.068481 26094 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:25.068710 26079 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:25.068804 26079 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "5f50a746500b4753b54d5cf1dfd9cfa1"
format_stamp: "Formatted at 2026-01-31 03:08:25 on dist-test-slave-5nnp"
server_key: "01c2911e47233bede52b8b4ecfc71e07"
server_key_iv: "923867bdb6dbc30668b7fa7468bf93fb"
server_key_version: "encryptionkey@0"
I20260131 03:08:25.068863 26079 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:25.084017 26079 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:25.086211 26079 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:25.086324 26079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:25.086575 26079 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:25.087204 26079 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:25.087256 26079 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:25.087281 26079 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:25.087294 26079 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:25.093125 26079 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:37685
I20260131 03:08:25.093199 26207 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:37685 every 8 connection(s)
I20260131 03:08:25.093541 26079 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
Jan 31 03:08:25 dist-test-slave-5nnp krb5kdc[25967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828905, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:08:25.100888 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 26079
I20260131 03:08:25.101023 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:08:25.101246 19792 external_mini_cluster.cc:1468] Setting key 2be8bb346d0911c7cf01a164e5ed342d
I20260131 03:08:25.103358 26210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:25.094252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33835 (local address 127.19.84.1:60599)
0131 03:08:25.094424 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:25.094458 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:25.094970 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:25.095236 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:25.095240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:25.095646 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:25.095967 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:25.095970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:25.096669 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:25.096670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:25.097090 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:25.097092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:25.097448 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:25.098028 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:25.098048 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:25.099765 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:25.101834 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:25.101846 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:25.101850 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:25.102185 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:25.102402 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:25.102403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:25.102403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:25.102459 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:25.102843 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:25.102849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:25.102983 (+ 134us) client_negotiation.cc:769] Sending connection context
0131 03:08:25.103089 (+ 106us) client_negotiation.cc:240] Negotiation successful
0131 03:08:25.103207 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260131 03:08:25.103443 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:25.094301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:60599 (local address 127.19.84.62:33835)
0131 03:08:25.094373 (+ 72us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:25.094376 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:25.094548 (+ 172us) server_negotiation.cc:408] Connection header received
0131 03:08:25.095075 (+ 527us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:25.095077 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:25.095088 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:25.095150 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:25.096061 (+ 911us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:25.096575 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:25.097173 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:25.097294 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:25.099885 (+ 2591us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:25.099891 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:25.099892 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:25.099900 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:25.101730 (+ 1830us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:25.102290 (+ 560us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:25.102292 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:25.102292 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:25.102324 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:25.102522 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:25.102523 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:25.102524 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:25.102744 (+ 220us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:25.102836 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:25.103132 (+ 296us) server_negotiation.cc:300] Negotiation successful
0131 03:08:25.103258 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260131 03:08:25.104589 26208 heartbeater.cc:344] Connected to a master server at 127.19.84.62:33835
I20260131 03:08:25.104729 26208 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:25.105007 26208 heartbeater.cc:507] Master 127.19.84.62:33835 requested a full tablet report, sending...
I20260131 03:08:25.105581 26015 ts_manager.cc:194] Registered new tserver with Master: 5f50a746500b4753b54d5cf1dfd9cfa1 (127.19.84.1:37685)
I20260131 03:08:25.106314 26015 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.1@KRBTEST.COM'} at 127.19.84.1:60599
WARNING: no policy specified for kudu/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:08:25.152491 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:33835
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:42613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:25.232885 26215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:25.233107 26215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:25.233150 26215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:25.234615 26215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:25.234697 26215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:25.234745 26215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:08:25.236557 26215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:42613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:33835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:25.236829 26215 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:25.237057 26215 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:25.240021 26220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:25.240013 26221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:25.240125 26215 server_base.cc:1060] running on GCE node
W20260131 03:08:25.240008 26223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:25.240655 26215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:25.240870 26215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:25.242029 26215 hybrid_clock.cc:648] HybridClock initialized: now 1769828905242006 us; error 39 us; skew 500 ppm
Jan 31 03:08:25 dist-test-slave-5nnp krb5kdc[25967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828905, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:25.244887 26215 init.cc:377] Logged in from keytab as kudu/127.19.84.2@KRBTEST.COM (short username kudu)
I20260131 03:08:25.245451 26215 webserver.cc:492] Webserver started at http://127.19.84.2:34085/ using document root <none> and password file <none>
I20260131 03:08:25.245656 26215 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:25.245711 26215 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:25.245854 26215 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:25.246899 26215 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "6abf211bbc754a00a16a9d10aa76763e"
format_stamp: "Formatted at 2026-01-31 03:08:25 on dist-test-slave-5nnp"
server_key: "27a6dd057f58f05367f3d942e274f423"
server_key_iv: "abef7bbf5e1e51b98ecf61e8156df884"
server_key_version: "encryptionkey@0"
I20260131 03:08:25.247227 26215 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "6abf211bbc754a00a16a9d10aa76763e"
format_stamp: "Formatted at 2026-01-31 03:08:25 on dist-test-slave-5nnp"
server_key: "27a6dd057f58f05367f3d942e274f423"
server_key_iv: "abef7bbf5e1e51b98ecf61e8156df884"
server_key_version: "encryptionkey@0"
I20260131 03:08:25.248608 26215 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:08:25.249440 26230 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:25.249711 26215 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:08:25.249823 26215 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "6abf211bbc754a00a16a9d10aa76763e"
format_stamp: "Formatted at 2026-01-31 03:08:25 on dist-test-slave-5nnp"
server_key: "27a6dd057f58f05367f3d942e274f423"
server_key_iv: "abef7bbf5e1e51b98ecf61e8156df884"
server_key_version: "encryptionkey@0"
I20260131 03:08:25.249919 26215 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:25.262696 26215 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:25.264856 26215 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:25.265115 26215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:25.265362 26215 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:25.265694 26215 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:25.265750 26215 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:25.265791 26215 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:25.265838 26215 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:25.272352 26215 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:39069
I20260131 03:08:25.272541 26343 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:39069 every 8 connection(s)
I20260131 03:08:25.273125 26215 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
I20260131 03:08:25.276922 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 26215
I20260131 03:08:25.277037 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:08:25.277245 19792 external_mini_cluster.cc:1468] Setting key 0d8cf72f5572da794dd9f368c85ede09
Jan 31 03:08:25 dist-test-slave-5nnp krb5kdc[25967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828905, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:08:25.283005 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:25.274012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:54333 (local address 127.19.84.62:33835)
0131 03:08:25.274082 (+ 70us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:25.274084 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:25.274322 (+ 238us) server_negotiation.cc:408] Connection header received
0131 03:08:25.274857 (+ 535us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:25.274858 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:25.274870 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:25.274930 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:25.275736 (+ 806us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:25.276259 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:25.277004 (+ 745us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:25.277183 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:25.280051 (+ 2868us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:25.280056 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:25.280057 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:25.280065 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:25.281489 (+ 1424us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:25.281925 (+ 436us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:25.281926 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:25.281926 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:25.281956 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:25.282154 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:25.282155 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:25.282156 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:25.282259 (+ 103us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:25.282333 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:25.282799 (+ 466us) server_negotiation.cc:300] Negotiation successful
0131 03:08:25.282908 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260131 03:08:25.283159 26346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:25.273985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33835 (local address 127.19.84.2:54333)
0131 03:08:25.274185 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:25.274215 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:25.274763 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:25.274993 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:25.274997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:25.275315 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:25.275646 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:25.275649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:25.276368 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:25.276369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:25.276884 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:25.276887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:25.277396 (+ 509us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:25.278103 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:25.278131 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:25.279954 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:25.281611 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:25.281618 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:25.281621 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:25.281849 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:25.282019 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:25.282020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:25.282021 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:25.282090 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:25.282321 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:25.282326 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:25.282464 (+ 138us) client_negotiation.cc:769] Sending connection context
0131 03:08:25.282781 (+ 317us) client_negotiation.cc:240] Negotiation successful
0131 03:08:25.282945 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":110,"threads_started":1}
I20260131 03:08:25.283779 26344 heartbeater.cc:344] Connected to a master server at 127.19.84.62:33835
I20260131 03:08:25.283913 26344 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:25.284133 26344 heartbeater.cc:507] Master 127.19.84.62:33835 requested a full tablet report, sending...
I20260131 03:08:25.284556 26015 ts_manager.cc:194] Registered new tserver with Master: 6abf211bbc754a00a16a9d10aa76763e (127.19.84.2:39069)
I20260131 03:08:25.284972 26015 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.2@KRBTEST.COM'} at 127.19.84.2:54333
WARNING: no policy specified for kudu/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:08:25.326529 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:33835
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:42613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:25.407120 26351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:25.407330 26351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:25.407368 26351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:25.408810 26351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:25.408882 26351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:25.408924 26351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:08:25.410521 26351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:42613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:33835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:25.410789 26351 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:25.410980 26351 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:25.413843 26359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:25.413921 26351 server_base.cc:1060] running on GCE node
W20260131 03:08:25.413843 26356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:25.413908 26357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:25.414175 26351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:25.414388 26351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:25.415568 26351 hybrid_clock.cc:648] HybridClock initialized: now 1769828905415553 us; error 26 us; skew 500 ppm
Jan 31 03:08:25 dist-test-slave-5nnp krb5kdc[25967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828905, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:25.417980 26351 init.cc:377] Logged in from keytab as kudu/127.19.84.3@KRBTEST.COM (short username kudu)
I20260131 03:08:25.418565 26351 webserver.cc:492] Webserver started at http://127.19.84.3:43859/ using document root <none> and password file <none>
I20260131 03:08:25.418772 26351 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:25.418828 26351 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:25.418960 26351 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:25.419991 26351 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "3511076eb90a48368befcda17a9974b8"
format_stamp: "Formatted at 2026-01-31 03:08:25 on dist-test-slave-5nnp"
server_key: "a457a159baa884593d4e23349c9ecb1b"
server_key_iv: "f5e139f3d188dd9d30da99a80d06d974"
server_key_version: "encryptionkey@0"
I20260131 03:08:25.420307 26351 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "3511076eb90a48368befcda17a9974b8"
format_stamp: "Formatted at 2026-01-31 03:08:25 on dist-test-slave-5nnp"
server_key: "a457a159baa884593d4e23349c9ecb1b"
server_key_iv: "f5e139f3d188dd9d30da99a80d06d974"
server_key_version: "encryptionkey@0"
I20260131 03:08:25.421531 26351 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:08:25.422250 26366 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:25.422468 26351 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:25.422587 26351 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "3511076eb90a48368befcda17a9974b8"
format_stamp: "Formatted at 2026-01-31 03:08:25 on dist-test-slave-5nnp"
server_key: "a457a159baa884593d4e23349c9ecb1b"
server_key_iv: "f5e139f3d188dd9d30da99a80d06d974"
server_key_version: "encryptionkey@0"
I20260131 03:08:25.422657 26351 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:25.438916 26351 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:25.440742 26351 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:25.440889 26351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:25.441112 26351 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:25.441421 26351 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:25.441473 26351 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:25.441516 26351 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:25.441552 26351 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:25.447252 26351 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:35043
I20260131 03:08:25.447301 26479 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:35043 every 8 connection(s)
I20260131 03:08:25.447643 26351 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
I20260131 03:08:25.451166 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 26351
I20260131 03:08:25.451268 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:08:25.451458 19792 external_mini_cluster.cc:1468] Setting key 8e7d8b739082ae731764091eb6b4e131
Jan 31 03:08:25 dist-test-slave-5nnp krb5kdc[25967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828905, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:08:25.456097 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:25.448326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:59205 (local address 127.19.84.62:33835)
0131 03:08:25.448394 (+ 68us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:25.448396 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:25.448570 (+ 174us) server_negotiation.cc:408] Connection header received
0131 03:08:25.449018 (+ 448us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:25.449019 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:25.449028 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:25.449080 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:25.449803 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:25.450294 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:25.450915 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:25.451069 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:25.453399 (+ 2330us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:25.453402 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:25.453402 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:25.453409 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:25.454942 (+ 1533us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:25.455314 (+ 372us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:25.455315 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:25.455316 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:25.455336 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:25.455578 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:25.455579 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:25.455579 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:25.455668 (+ 89us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:25.455722 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:25.455959 (+ 237us) server_negotiation.cc:300] Negotiation successful
0131 03:08:25.456031 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260131 03:08:25.456175 26482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:25.448290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33835 (local address 127.19.84.3:59205)
0131 03:08:25.448453 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:25.448479 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:25.448944 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:25.449144 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:25.449147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:25.449440 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:25.449720 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:25.449723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:25.450360 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:25.450360 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:25.450820 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:25.450822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:25.451294 (+ 472us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:25.451857 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:25.451885 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:25.453304 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:25.455038 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:25.455043 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:25.455045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:25.455243 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:25.455447 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:25.455448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:25.455448 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:25.455519 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:25.455730 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:25.455733 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:25.455843 (+ 110us) client_negotiation.cc:769] Sending connection context
0131 03:08:25.455931 (+ 88us) client_negotiation.cc:240] Negotiation successful
0131 03:08:25.456038 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260131 03:08:25.456696 26480 heartbeater.cc:344] Connected to a master server at 127.19.84.62:33835
I20260131 03:08:25.456827 26480 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:25.457067 26480 heartbeater.cc:507] Master 127.19.84.62:33835 requested a full tablet report, sending...
I20260131 03:08:25.457594 26015 ts_manager.cc:194] Registered new tserver with Master: 3511076eb90a48368befcda17a9974b8 (127.19.84.3:35043)
I20260131 03:08:25.457957 26015 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:59205
I20260131 03:08:25.463243 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/test-admin.keytab.
Jan 31 03:08:25 dist-test-slave-5nnp krb5kdc[25967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:25.479497 19792 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260131 03:08:25.481316 26487 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:25.480503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53686 (local address 127.19.84.1:37685)
0131 03:08:25.480692 (+ 189us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:25.480700 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:25.480716 (+ 16us) server_negotiation.cc:408] Connection header received
0131 03:08:25.480751 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:25.480753 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:25.480767 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:25.480832 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:25.481222 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53686: BlockingRecv error: recv got EOF from 127.0.0.1:53686 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":110,"threads_started":1}
W20260131 03:08:25.481376 19792 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/krb5cc)
Jan 31 03:08:25 dist-test-slave-5nnp krb5kdc[25967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:25.482422 19792 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jan 31 03:08:25 dist-test-slave-5nnp krb5kdc[25967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:25.483507 19792 init.cc:295] Successfully reacquired a new kerberos TGT
Jan 31 03:08:25 dist-test-slave-5nnp krb5kdc[25967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.1@KRBTEST.COM
I20260131 03:08:25.490515 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:25.484156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53688 (local address 127.19.84.1:37685)
0131 03:08:25.484211 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:25.484213 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:25.484226 (+ 13us) server_negotiation.cc:408] Connection header received
0131 03:08:25.484295 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:25.484295 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:25.484306 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:25.484356 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:25.484894 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:25.485656 (+ 762us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:25.486224 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:25.486465 (+ 241us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:25.488227 (+ 1762us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:25.488236 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:25.488237 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:25.488242 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:25.489629 (+ 1387us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:25.489941 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:25.489943 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:25.489943 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:25.489974 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:25.490099 (+ 125us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:25.490100 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:25.490100 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:25.490183 (+ 83us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:25.490283 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:25.490370 (+ 87us) server_negotiation.cc:300] Negotiation successful
0131 03:08:25.490436 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260131 03:08:25.492339 26487 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:25.491862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53694 (local address 127.19.84.1:37685)
0131 03:08:25.491918 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:25.491919 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:25.491926 (+ 7us) server_negotiation.cc:408] Connection header received
0131 03:08:25.491984 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:25.491985 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:25.491993 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:25.492038 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:25.492303 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53694: BlockingRecv error: recv got EOF from 127.0.0.1:53694 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260131 03:08:25.492434 19792 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/krb5cc' of type FILE
Jan 31 03:08:25 dist-test-slave-5nnp krb5kdc[25967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:25.493342 19792 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jan 31 03:08:25 dist-test-slave-5nnp krb5kdc[25967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:25.494143 19792 init.cc:295] Successfully reacquired a new kerberos TGT
Jan 31 03:08:25 dist-test-slave-5nnp krb5kdc[25967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.1@KRBTEST.COM
I20260131 03:08:25.500555 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:25.494680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53704 (local address 127.19.84.1:37685)
0131 03:08:25.494750 (+ 70us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:25.494751 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:25.494786 (+ 35us) server_negotiation.cc:408] Connection header received
0131 03:08:25.494864 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:25.494864 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:25.494872 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:25.494918 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:25.495348 (+ 430us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:25.495912 (+ 564us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:25.496396 (+ 484us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:25.496557 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:25.498216 (+ 1659us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:25.498219 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:25.498219 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:25.498225 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:25.499734 (+ 1509us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:25.500000 (+ 266us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:25.500001 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:25.500001 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:25.500021 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:25.500131 (+ 110us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:25.500132 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:25.500132 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:25.500195 (+ 63us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:25.500261 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:25.500338 (+ 77us) server_negotiation.cc:300] Negotiation successful
0131 03:08:25.500398 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260131 03:08:25.502117 26487 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:25.501593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53718 (local address 127.19.84.1:37685)
0131 03:08:25.501652 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:25.501654 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:25.501694 (+ 40us) server_negotiation.cc:408] Connection header received
0131 03:08:25.501753 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:25.501754 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:25.501763 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:25.501806 (+ 43us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:25.502069 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53718: BlockingRecv error: recv got EOF from 127.0.0.1:53718 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260131 03:08:25.502230 19792 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769828849314684-19792-0/krb5kdc/krb5cc' of type FILE
Jan 31 03:08:25 dist-test-slave-5nnp krb5kdc[25967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:25.503151 19792 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jan 31 03:08:25 dist-test-slave-5nnp krb5kdc[25967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:25.504099 19792 init.cc:295] Successfully reacquired a new kerberos TGT
Jan 31 03:08:25 dist-test-slave-5nnp krb5kdc[25967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.1@KRBTEST.COM
I20260131 03:08:25.510421 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:25.504607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53722 (local address 127.19.84.1:37685)
0131 03:08:25.504664 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:25.504666 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:25.504676 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:08:25.504776 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:25.504777 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:25.504784 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:25.504845 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:25.505311 (+ 466us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:25.505758 (+ 447us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:25.506277 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:25.506454 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:25.508169 (+ 1715us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:25.508173 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:25.508174 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:25.508181 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:25.509381 (+ 1200us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:25.509764 (+ 383us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:25.509765 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:25.509765 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:25.509796 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:25.509945 (+ 149us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:25.509946 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:25.509946 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:25.510026 (+ 80us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:25.510141 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:25.510236 (+ 95us) server_negotiation.cc:300] Negotiation successful
0131 03:08:25.510316 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260131 03:08:25.510988 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 26079
I20260131 03:08:25.516259 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 26215
I20260131 03:08:25.520531 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 26351
I20260131 03:08:25.524806 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 25983
2026-01-31T03:08:25Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2905 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:08:25 dist-test-slave-5nnp krb5kdc[26500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:08:25 dist-test-slave-5nnp krb5kdc[26500](info): set up 2 sockets
Jan 31 03:08:25 dist-test-slave-5nnp krb5kdc[26500](info): commencing operation
krb5kdc: starting...
W20260131 03:08:27.544142 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.999s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:08:27 dist-test-slave-5nnp krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:08:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:08:27Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab.
I20260131 03:08:27.678021 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:33805
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:37697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:33805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:27.757180 26516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:27.757426 26516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:27.757465 26516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:27.759084 26516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:08:27.759155 26516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:27.759176 26516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:27.759197 26516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:08:27.759225 26516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:08:27.760921 26516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:37697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:33805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:33805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:27.761206 26516 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:27.761409 26516 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:27.764052 26521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:27.764125 26516 server_base.cc:1060] running on GCE node
W20260131 03:08:27.764052 26522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:27.764249 26524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:27.764531 26516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:27.764742 26516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:27.765893 26516 hybrid_clock.cc:648] HybridClock initialized: now 1769828907765875 us; error 38 us; skew 500 ppm
Jan 31 03:08:27 dist-test-slave-5nnp krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828907, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:27.768929 26516 init.cc:377] Logged in from keytab as oryx/127.19.84.62@KRBTEST.COM (short username oryx)
I20260131 03:08:27.769685 26516 webserver.cc:492] Webserver started at http://127.19.84.62:41907/ using document root <none> and password file <none>
I20260131 03:08:27.769985 26516 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:27.770046 26516 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:27.770164 26516 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:27.771523 26516 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "6c41b3e9bc84467e9d9758cbd2691e72"
format_stamp: "Formatted at 2026-01-31 03:08:27 on dist-test-slave-5nnp"
server_key: "ed4f32010e384267316abb9a51e14913"
server_key_iv: "3362fa7f9596a1c0c3fa36f5bcdce876"
server_key_version: "encryptionkey@0"
I20260131 03:08:27.772154 26516 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "6c41b3e9bc84467e9d9758cbd2691e72"
format_stamp: "Formatted at 2026-01-31 03:08:27 on dist-test-slave-5nnp"
server_key: "ed4f32010e384267316abb9a51e14913"
server_key_iv: "3362fa7f9596a1c0c3fa36f5bcdce876"
server_key_version: "encryptionkey@0"
I20260131 03:08:27.774058 26516 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260131 03:08:27.777918 26531 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:27.778790 26516 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260131 03:08:27.779003 26516 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "6c41b3e9bc84467e9d9758cbd2691e72"
format_stamp: "Formatted at 2026-01-31 03:08:27 on dist-test-slave-5nnp"
server_key: "ed4f32010e384267316abb9a51e14913"
server_key_iv: "3362fa7f9596a1c0c3fa36f5bcdce876"
server_key_version: "encryptionkey@0"
I20260131 03:08:27.779089 26516 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:27.792567 26516 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:27.798417 26516 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:27.798686 26516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:27.803958 26516 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:33805
I20260131 03:08:27.804008 26583 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:33805 every 8 connection(s)
I20260131 03:08:27.804524 26516 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:08:27.805362 26584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:27.809535 26584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72: Bootstrap starting.
I20260131 03:08:27.810276 26584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:27.810519 26584 log.cc:826] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:27.811295 26584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72: No bootstrap required, opened a new log
I20260131 03:08:27.812534 26584 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } }
I20260131 03:08:27.812697 26584 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:27.812732 26584 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c41b3e9bc84467e9d9758cbd2691e72, State: Initialized, Role: FOLLOWER
I20260131 03:08:27.812803 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 26516
I20260131 03:08:27.812916 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:08:27.812933 26584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } }
I20260131 03:08:27.813019 26584 raft_consensus.cc:399] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:08:27.813056 26584 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:08:27.813078 19792 external_mini_cluster.cc:1468] Setting key c765182b2412684d1b4091b07bcb6339
I20260131 03:08:27.813107 26584 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:27.813632 26584 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } }
I20260131 03:08:27.813772 26584 leader_election.cc:304] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c41b3e9bc84467e9d9758cbd2691e72; no voters:
I20260131 03:08:27.813946 26584 leader_election.cc:290] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:08:27.813978 26589 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:08:27.814074 26589 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [term 1 LEADER]: Becoming Leader. State: Replica: 6c41b3e9bc84467e9d9758cbd2691e72, State: Running, Role: LEADER
I20260131 03:08:27.814150 26589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } }
I20260131 03:08:27.814203 26584 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:08:27.814446 26589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c41b3e9bc84467e9d9758cbd2691e72. Latest consensus state: current_term: 1 leader_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } }
I20260131 03:08:27.814499 26589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:27.814718 26593 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:08:27.815361 26593 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:08:27.816110 26591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } }
I20260131 03:08:27.817618 26593 catalog_manager.cc:1357] Generated new cluster ID: 00b8836497954b1f8f0752703e43dd26
I20260131 03:08:27.818915 26591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:27.818950 26593 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Jan 31 03:08:27 dist-test-slave-5nnp krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM
I20260131 03:08:27.824052 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:27.813747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58364 (local address 127.19.84.62:33805)
0131 03:08:27.813949 (+ 202us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:27.813965 (+ 16us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:27.814004 (+ 39us) server_negotiation.cc:408] Connection header received
0131 03:08:27.814124 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:27.814131 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:27.814183 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:27.814304 (+ 121us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:27.815507 (+ 1203us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:27.817511 (+ 2004us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:27.818236 (+ 725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:27.818491 (+ 255us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:27.820712 (+ 2221us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:27.820724 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:27.820724 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:27.820743 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:27.822440 (+ 1697us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:27.822837 (+ 397us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:27.822840 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:27.822841 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:27.822909 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:27.823075 (+ 166us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:27.823076 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:27.823076 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:27.823265 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:27.823474 (+ 209us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:27.823688 (+ 214us) server_negotiation.cc:300] Negotiation successful
0131 03:08:27.823919 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":89,"threads_started":1}
I20260131 03:08:27.863994 26593 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:08:27.864820 26593 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:08:27.876946 26593 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72: Generated new TSK 0
I20260131 03:08:27.877158 26593 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab.
I20260131 03:08:27.935424 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:33805
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:37697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:28.016601 26612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:28.016824 26612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:28.016865 26612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:28.018349 26612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:28.018424 26612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:28.018468 26612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:08:28.020262 26612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:37697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:33805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:28.020527 26612 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:28.020732 26612 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:28.023484 26618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:28.023479 26620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:28.023584 26617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:28.023644 26612 server_base.cc:1060] running on GCE node
I20260131 03:08:28.023953 26612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:28.024184 26612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:28.025400 26612 hybrid_clock.cc:648] HybridClock initialized: now 1769828908025349 us; error 43 us; skew 500 ppm
Jan 31 03:08:28 dist-test-slave-5nnp krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828908, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:28.030372 26612 init.cc:377] Logged in from keytab as oryx/127.19.84.1@KRBTEST.COM (short username oryx)
I20260131 03:08:28.031097 26612 webserver.cc:492] Webserver started at http://127.19.84.1:34329/ using document root <none> and password file <none>
I20260131 03:08:28.031297 26612 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:28.031330 26612 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:28.031466 26612 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:28.033527 26612 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "0df52ee4e3d4496baaf359d610b7d34c"
format_stamp: "Formatted at 2026-01-31 03:08:28 on dist-test-slave-5nnp"
server_key: "55abfd4d9937bcda1a06e3abd5878b99"
server_key_iv: "8d918f58c3a661b761b5cf83b788c569"
server_key_version: "encryptionkey@0"
I20260131 03:08:28.034883 26612 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "0df52ee4e3d4496baaf359d610b7d34c"
format_stamp: "Formatted at 2026-01-31 03:08:28 on dist-test-slave-5nnp"
server_key: "55abfd4d9937bcda1a06e3abd5878b99"
server_key_iv: "8d918f58c3a661b761b5cf83b788c569"
server_key_version: "encryptionkey@0"
I20260131 03:08:28.036686 26612 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260131 03:08:28.037465 26627 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:28.037670 26612 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260131 03:08:28.037736 26612 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "0df52ee4e3d4496baaf359d610b7d34c"
format_stamp: "Formatted at 2026-01-31 03:08:28 on dist-test-slave-5nnp"
server_key: "55abfd4d9937bcda1a06e3abd5878b99"
server_key_iv: "8d918f58c3a661b761b5cf83b788c569"
server_key_version: "encryptionkey@0"
I20260131 03:08:28.037834 26612 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:28.052101 26612 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:28.054119 26612 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:28.054258 26612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:28.054502 26612 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:28.054831 26612 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:28.054863 26612 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:28.054886 26612 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:28.054932 26612 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:28.060932 26612 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:35541
I20260131 03:08:28.060986 26740 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:35541 every 8 connection(s)
I20260131 03:08:28.061295 26612 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
Jan 31 03:08:28 dist-test-slave-5nnp krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828908, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM
I20260131 03:08:28.069389 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.062052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:46021 (local address 127.19.84.62:33805)
0131 03:08:28.062116 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:28.062118 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:28.062273 (+ 155us) server_negotiation.cc:408] Connection header received
0131 03:08:28.062712 (+ 439us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:28.062714 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:28.062727 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:28.062792 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:28.063619 (+ 827us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.064125 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.064679 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.064793 (+ 114us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.067028 (+ 2235us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:28.067032 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:28.067032 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:28.067037 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:28.068352 (+ 1315us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.068690 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.068690 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.068691 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.068711 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.068898 (+ 187us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.068898 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.068898 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.068982 (+ 84us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:28.069031 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:28.069243 (+ 212us) server_negotiation.cc:300] Negotiation successful
0131 03:08:28.069320 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260131 03:08:28.069494 26743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.062000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.1:46021)
0131 03:08:28.062155 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:28.062190 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:28.062641 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:28.062850 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:28.062853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:28.063140 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:28.063506 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.063518 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.064215 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.064216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:28.064608 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.064610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.064712 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.065592 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:28.065608 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:28.066950 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:28.068427 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.068432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.068434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.068625 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.068775 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.068776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.068776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.068848 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.069031 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:28.069035 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:28.069138 (+ 103us) client_negotiation.cc:769] Sending connection context
0131 03:08:28.069221 (+ 83us) client_negotiation.cc:240] Negotiation successful
0131 03:08:28.069317 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260131 03:08:28.070050 26741 heartbeater.cc:344] Connected to a master server at 127.19.84.62:33805
I20260131 03:08:28.070129 26741 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:28.070355 26741 heartbeater.cc:507] Master 127.19.84.62:33805 requested a full tablet report, sending...
I20260131 03:08:28.070562 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 26612
I20260131 03:08:28.070685 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:08:28.070838 19792 external_mini_cluster.cc:1468] Setting key 7f81d767b31d96f0302cc981ffada1b3
I20260131 03:08:28.070894 26541 ts_manager.cc:194] Registered new tserver with Master: 0df52ee4e3d4496baaf359d610b7d34c (127.19.84.1:35541)
I20260131 03:08:28.071672 26541 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.84.1@KRBTEST.COM'} at 127.19.84.1:46021
WARNING: no policy specified for oryx/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab.
I20260131 03:08:28.113940 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:33805
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:37697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:28.192443 26748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:28.192799 26748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:28.192839 26748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:28.194348 26748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:28.194423 26748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:28.194469 26748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:08:28.196125 26748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:37697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:33805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:28.196398 26748 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:28.196611 26748 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:28.199137 26756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:28.199138 26754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:28.199162 26753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:28.199349 26748 server_base.cc:1060] running on GCE node
I20260131 03:08:28.199554 26748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:28.199715 26748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:28.200845 26748 hybrid_clock.cc:648] HybridClock initialized: now 1769828908200832 us; error 28 us; skew 500 ppm
Jan 31 03:08:28 dist-test-slave-5nnp krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828908, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:28.203162 26748 init.cc:377] Logged in from keytab as oryx/127.19.84.2@KRBTEST.COM (short username oryx)
I20260131 03:08:28.203778 26748 webserver.cc:492] Webserver started at http://127.19.84.2:37775/ using document root <none> and password file <none>
I20260131 03:08:28.203987 26748 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:28.204023 26748 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:28.204146 26748 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:28.205206 26748 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6"
format_stamp: "Formatted at 2026-01-31 03:08:28 on dist-test-slave-5nnp"
server_key: "a344c4481a47f0f454483959b448195b"
server_key_iv: "96800618cf59505e9e470cc86fee9090"
server_key_version: "encryptionkey@0"
I20260131 03:08:28.205559 26748 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6"
format_stamp: "Formatted at 2026-01-31 03:08:28 on dist-test-slave-5nnp"
server_key: "a344c4481a47f0f454483959b448195b"
server_key_iv: "96800618cf59505e9e470cc86fee9090"
server_key_version: "encryptionkey@0"
I20260131 03:08:28.206784 26748 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:08:28.207490 26763 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:28.207715 26748 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:28.207813 26748 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6"
format_stamp: "Formatted at 2026-01-31 03:08:28 on dist-test-slave-5nnp"
server_key: "a344c4481a47f0f454483959b448195b"
server_key_iv: "96800618cf59505e9e470cc86fee9090"
server_key_version: "encryptionkey@0"
I20260131 03:08:28.207895 26748 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:28.229439 26748 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:28.231156 26748 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:28.231320 26748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:28.231600 26748 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:28.231940 26748 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:28.231997 26748 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:28.232036 26748 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:28.232066 26748 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:28.239122 26748 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:39245
I20260131 03:08:28.239161 26876 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:39245 every 8 connection(s)
I20260131 03:08:28.239578 26748 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
Jan 31 03:08:28 dist-test-slave-5nnp krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828908, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM
I20260131 03:08:28.248318 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 26748
I20260131 03:08:28.248435 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:08:28.248607 19792 external_mini_cluster.cc:1468] Setting key 896eee62306ddade7e6213739e623371
I20260131 03:08:28.248669 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.240291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:38809 (local address 127.19.84.62:33805)
0131 03:08:28.240361 (+ 70us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:28.240363 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:28.240533 (+ 170us) server_negotiation.cc:408] Connection header received
0131 03:08:28.241135 (+ 602us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:28.241136 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:28.241150 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:28.241195 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:28.242091 (+ 896us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.242593 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.243169 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.243340 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.245743 (+ 2403us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:28.245749 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:28.245749 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:28.245758 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:28.247181 (+ 1423us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.247722 (+ 541us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.247723 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.247723 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.247757 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.247998 (+ 241us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.247999 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.248000 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.248101 (+ 101us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:28.248204 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:28.248472 (+ 268us) server_negotiation.cc:300] Negotiation successful
0131 03:08:28.248574 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260131 03:08:28.248857 26879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.240249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.2:38809)
0131 03:08:28.240406 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:28.240441 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:28.241022 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:28.241288 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:28.241292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:28.241692 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:28.241984 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.241987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.242688 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.242689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:28.243080 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.243082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.243241 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.244244 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:28.244262 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:28.245651 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:28.247324 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.247330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.247333 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.247634 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.247825 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.247826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.247826 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.247915 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.248195 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:28.248200 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:28.248333 (+ 133us) client_negotiation.cc:769] Sending connection context
0131 03:08:28.248442 (+ 109us) client_negotiation.cc:240] Negotiation successful
0131 03:08:28.248582 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260131 03:08:28.249461 26877 heartbeater.cc:344] Connected to a master server at 127.19.84.62:33805
I20260131 03:08:28.249580 26877 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:28.249800 26877 heartbeater.cc:507] Master 127.19.84.62:33805 requested a full tablet report, sending...
I20260131 03:08:28.250188 26541 ts_manager.cc:194] Registered new tserver with Master: 33d5b3c3b9d74484b9a5f92b2dc57fa6 (127.19.84.2:39245)
I20260131 03:08:28.250579 26541 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.84.2@KRBTEST.COM'} at 127.19.84.2:38809
WARNING: no policy specified for oryx/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab.
I20260131 03:08:28.294431 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:33805
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:37697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:28.373500 26884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:28.373687 26884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:28.373710 26884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:28.375185 26884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:28.375231 26884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:28.375285 26884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:08:28.377089 26884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:37697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:33805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:28.377331 26884 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:28.377542 26884 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:28.380199 26889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:28.380200 26892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:28.380219 26890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:28.380286 26884 server_base.cc:1060] running on GCE node
I20260131 03:08:28.380595 26884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:28.380839 26884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:28.381992 26884 hybrid_clock.cc:648] HybridClock initialized: now 1769828908381993 us; error 39 us; skew 500 ppm
Jan 31 03:08:28 dist-test-slave-5nnp krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828908, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:28.384348 26884 init.cc:377] Logged in from keytab as oryx/127.19.84.3@KRBTEST.COM (short username oryx)
I20260131 03:08:28.384907 26884 webserver.cc:492] Webserver started at http://127.19.84.3:43883/ using document root <none> and password file <none>
I20260131 03:08:28.385066 26884 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:28.385097 26884 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:28.385175 26884 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:28.386056 26884 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "e174c1a5c18148c5ac26d4824b84fe49"
format_stamp: "Formatted at 2026-01-31 03:08:28 on dist-test-slave-5nnp"
server_key: "0df258b4289d04243c2088951e8e82d2"
server_key_iv: "d56e26f0e6ddc237e482fd61454e74f4"
server_key_version: "encryptionkey@0"
I20260131 03:08:28.386364 26884 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "e174c1a5c18148c5ac26d4824b84fe49"
format_stamp: "Formatted at 2026-01-31 03:08:28 on dist-test-slave-5nnp"
server_key: "0df258b4289d04243c2088951e8e82d2"
server_key_iv: "d56e26f0e6ddc237e482fd61454e74f4"
server_key_version: "encryptionkey@0"
I20260131 03:08:28.387517 26884 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:08:28.388176 26899 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:28.388417 26884 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:28.388473 26884 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "e174c1a5c18148c5ac26d4824b84fe49"
format_stamp: "Formatted at 2026-01-31 03:08:28 on dist-test-slave-5nnp"
server_key: "0df258b4289d04243c2088951e8e82d2"
server_key_iv: "d56e26f0e6ddc237e482fd61454e74f4"
server_key_version: "encryptionkey@0"
I20260131 03:08:28.388518 26884 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:28.401899 26884 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:28.403604 26884 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:28.403707 26884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:28.403889 26884 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:28.404186 26884 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:28.404217 26884 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:28.404239 26884 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:28.404287 26884 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:28.409934 26884 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:44133
I20260131 03:08:28.410027 27012 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:44133 every 8 connection(s)
I20260131 03:08:28.410312 26884 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
Jan 31 03:08:28 dist-test-slave-5nnp krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828908, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM
I20260131 03:08:28.418859 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 26884
I20260131 03:08:28.418936 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:08:28.419090 19792 external_mini_cluster.cc:1468] Setting key 27d8729e02b72e0e160aa2bf34a4a8f8
I20260131 03:08:28.419658 27015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.411070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.3:48439)
0131 03:08:28.411312 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:28.411337 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:28.411873 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:28.412120 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:28.412124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:28.412443 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:28.412729 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.412733 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.413429 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.413430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:28.413821 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.413822 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.413970 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.415038 (+ 1068us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:28.415056 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:28.416702 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:28.418220 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.418226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.418229 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.418434 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.418608 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.418609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.418609 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.418710 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.418991 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:28.418998 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:28.419124 (+ 126us) client_negotiation.cc:769] Sending connection context
0131 03:08:28.419252 (+ 128us) client_negotiation.cc:240] Negotiation successful
0131 03:08:28.419420 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":110,"threads_started":1}
I20260131 03:08:28.419651 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.411120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:48439 (local address 127.19.84.62:33805)
0131 03:08:28.411248 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:28.411251 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:28.411446 (+ 195us) server_negotiation.cc:408] Connection header received
0131 03:08:28.411972 (+ 526us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:28.411974 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:28.411986 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:28.412051 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:28.412841 (+ 790us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.413336 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.413917 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.414086 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.416781 (+ 2695us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:28.416787 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:28.416787 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:28.416803 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:28.418136 (+ 1333us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.418511 (+ 375us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.418512 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.418513 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.418543 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.418783 (+ 240us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.418784 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.418785 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.418883 (+ 98us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:28.418964 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:28.419413 (+ 449us) server_negotiation.cc:300] Negotiation successful
0131 03:08:28.419519 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260131 03:08:28.420203 27013 heartbeater.cc:344] Connected to a master server at 127.19.84.62:33805
I20260131 03:08:28.420291 27013 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:28.420498 27013 heartbeater.cc:507] Master 127.19.84.62:33805 requested a full tablet report, sending...
I20260131 03:08:28.420884 26541 ts_manager.cc:194] Registered new tserver with Master: e174c1a5c18148c5ac26d4824b84fe49 (127.19.84.3:44133)
I20260131 03:08:28.421262 26541 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:48439
I20260131 03:08:28.431581 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.19.84.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.84.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.84.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.19.84.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.19.84.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.19.84.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab.
I20260131 03:08:28.476037 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.61:38999
--webserver_interface=127.19.84.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:37697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:33805,127.19.84.61:38999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:28.554966 27020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:28.555151 27020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:28.555174 27020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:28.556728 27020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:08:28.556779 27020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:28.556799 27020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:28.556813 27020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:08:28.556869 27020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:08:28.558439 27020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:37697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:33805,127.19.84.61:38999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.61:38999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.84.61
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
W20260131 03:08:28.558569 27020 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.19.84.62:33805,127.19.84.61:38999'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260131 03:08:28.558717 27020 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:28.558917 27020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:28.561293 27028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:28.561299 27026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:28.561321 27025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:28.561592 27020 server_base.cc:1060] running on GCE node
I20260131 03:08:28.561838 27020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:28.562078 27020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:28.563212 27020 hybrid_clock.cc:648] HybridClock initialized: now 1769828908563198 us; error 44 us; skew 500 ppm
Jan 31 03:08:28 dist-test-slave-5nnp krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828908, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:28.565961 27020 init.cc:377] Logged in from keytab as oryx/127.19.84.61@KRBTEST.COM (short username oryx)
I20260131 03:08:28.566646 27020 webserver.cc:492] Webserver started at http://127.19.84.61:44789/ using document root <none> and password file <none>
I20260131 03:08:28.566885 27020 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:28.566941 27020 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:28.567060 27020 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:28.568315 27020 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/data/instance:
uuid: "b75c66249f3f4d7d8a97151e41f9f403"
format_stamp: "Formatted at 2026-01-31 03:08:28 on dist-test-slave-5nnp"
server_key: "64d594dd0acf749838672ee8aa8137f0"
server_key_iv: "88fe832738e828c7e30c0b3c5f193bd9"
server_key_version: "encryptionkey@0"
I20260131 03:08:28.568619 27020 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/wal/instance:
uuid: "b75c66249f3f4d7d8a97151e41f9f403"
format_stamp: "Formatted at 2026-01-31 03:08:28 on dist-test-slave-5nnp"
server_key: "64d594dd0acf749838672ee8aa8137f0"
server_key_iv: "88fe832738e828c7e30c0b3c5f193bd9"
server_key_version: "encryptionkey@0"
I20260131 03:08:28.569837 27020 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:08:28.570459 27035 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:28.570668 27020 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260131 03:08:28.570755 27020 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/wal
uuid: "b75c66249f3f4d7d8a97151e41f9f403"
format_stamp: "Formatted at 2026-01-31 03:08:28 on dist-test-slave-5nnp"
server_key: "64d594dd0acf749838672ee8aa8137f0"
server_key_iv: "88fe832738e828c7e30c0b3c5f193bd9"
server_key_version: "encryptionkey@0"
I20260131 03:08:28.570837 27020 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:28.579998 27020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:28.581615 27020 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:28.581766 27020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:28.585618 27020 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.61:38999
I20260131 03:08:28.585654 27087 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.61:38999 every 8 connection(s)
I20260131 03:08:28.585950 27020 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/data/info.pb
I20260131 03:08:28.586553 27088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:28.588382 27088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } has no permanent_uuid. Determining permanent_uuid...
I20260131 03:08:28.590052 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 27020
Jan 31 03:08:28 dist-test-slave-5nnp krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828908, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.61@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM
Jan 31 03:08:28 dist-test-slave-5nnp krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.84.61@KRBTEST.COM
I20260131 03:08:28.596889 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.589043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58370 (local address 127.19.84.62:33805)
0131 03:08:28.589130 (+ 87us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:28.589132 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:28.589264 (+ 132us) server_negotiation.cc:408] Connection header received
0131 03:08:28.589653 (+ 389us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:28.589654 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:28.589667 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:28.589736 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:28.590705 (+ 969us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.591202 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.591836 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.592038 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.593938 (+ 1900us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:28.593943 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:28.593944 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:28.593951 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:28.595335 (+ 1384us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.595864 (+ 529us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.595865 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.595866 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.595893 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.596130 (+ 237us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.596130 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.596131 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.596287 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:28.596368 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:28.596682 (+ 314us) server_negotiation.cc:300] Negotiation successful
0131 03:08:28.596773 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260131 03:08:28.596951 27090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.588973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.0.0.1:58370)
0131 03:08:28.589159 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:28.589191 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:28.589549 (+ 358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:28.589811 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:28.589814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:28.590228 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:28.590583 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.590587 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.591303 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.591304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:28.591743 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.591745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.591884 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.592428 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:28.592443 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:28.593852 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:28.595524 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.595530 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.595533 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.595794 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.595982 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.595983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.595983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.596058 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.596361 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:28.596365 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:28.596477 (+ 112us) client_negotiation.cc:769] Sending connection context
0131 03:08:28.596637 (+ 160us) client_negotiation.cc:240] Negotiation successful
0131 03:08:28.596798 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
I20260131 03:08:28.597553 27088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } has no permanent_uuid. Determining permanent_uuid...
I20260131 03:08:28.599140 27092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.590559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34642 (local address 127.19.84.61:38999)
0131 03:08:28.590698 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:28.590706 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:28.590725 (+ 19us) server_negotiation.cc:408] Connection header received
0131 03:08:28.590800 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:28.590803 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:28.590820 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:28.590901 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:28.591645 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.592617 (+ 972us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.593186 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.593395 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.595731 (+ 2336us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:28.595737 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:28.595738 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:28.595743 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:28.598077 (+ 2334us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.598382 (+ 305us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.598385 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.598385 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.598422 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.598612 (+ 190us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.598612 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.598613 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.598723 (+ 110us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:28.598874 (+ 151us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:28.598943 (+ 69us) server_negotiation.cc:300] Negotiation successful
0131 03:08:28.599026 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
Jan 31 03:08:28 dist-test-slave-5nnp krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828908, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.61@KRBTEST.COM for oryx/127.19.84.61@KRBTEST.COM
I20260131 03:08:28.603281 27090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.597779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:38999 (local address 127.0.0.1:34648)
0131 03:08:28.597823 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:28.597833 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:28.597872 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:28.598084 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:28.598085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:28.598232 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:28.598384 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.598385 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.599006 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.599008 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:28.599507 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.599509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.599599 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.600032 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:28.600035 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:28.601076 (+ 1041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:28.602471 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.602472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.602472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.602609 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.602784 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.602785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.602785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.602810 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.602999 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:28.602999 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:28.603031 (+ 32us) client_negotiation.cc:769] Sending connection context
0131 03:08:28.603099 (+ 68us) client_negotiation.cc:240] Negotiation successful
0131 03:08:28.603185 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260131 03:08:28.603363 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.597842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34648 (local address 127.19.84.61:38999)
0131 03:08:28.597969 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:28.597971 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:28.597978 (+ 7us) server_negotiation.cc:408] Connection header received
0131 03:08:28.597989 (+ 11us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:28.597990 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:28.597996 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:28.598037 (+ 41us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:28.598460 (+ 423us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.598925 (+ 465us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.599671 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.599862 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.601159 (+ 1297us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:28.601164 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:28.601168 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:28.601174 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:28.602380 (+ 1206us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.602688 (+ 308us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.602689 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.602689 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.602712 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.602863 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.602864 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.602864 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.602939 (+ 75us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:28.603008 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:28.603122 (+ 114us) server_negotiation.cc:300] Negotiation successful
0131 03:08:28.603250 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":115,"thread_start_us":91,"threads_started":1}
I20260131 03:08:28.604738 27088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403: Bootstrap starting.
I20260131 03:08:28.605237 27088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:28.605440 27088 log.cc:826] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:28.606071 27088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403: No bootstrap required, opened a new log
I20260131 03:08:28.607074 27088 raft_consensus.cc:359] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } }
I20260131 03:08:28.607172 27088 raft_consensus.cc:385] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:28.607192 27088 raft_consensus.cc:740] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75c66249f3f4d7d8a97151e41f9f403, State: Initialized, Role: FOLLOWER
I20260131 03:08:28.607256 27088 consensus_queue.cc:260] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } }
I20260131 03:08:28.607481 27095 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } } }
I20260131 03:08:28.607525 27088 sys_catalog.cc:565] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:08:28.607570 27095 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [sys.catalog]: This master's current role is: FOLLOWER
W20260131 03:08:28.609263 27106 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:08:28.609321 27106 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:08:28.610554 27020 master_runner.cc:428] Detected that this master b75c66249f3f4d7d8a97151e41f9f403 is joining an existing cluster
I20260131 03:08:28.610600 27020 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.84.61:38999
I20260131 03:08:28.617717 27112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.611632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:38999 (local address 127.0.0.1:34660)
0131 03:08:28.611780 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:28.611801 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:28.611890 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:28.612061 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:28.612062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:28.612325 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:28.612849 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.612851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.613597 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.613602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:28.613970 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.613972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.614469 (+ 497us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.614881 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:28.614885 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:28.615229 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:28.616836 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.616837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.616837 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.616974 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.617144 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.617145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.617146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.617172 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.617401 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:28.617402 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:28.617435 (+ 33us) client_negotiation.cc:769] Sending connection context
0131 03:08:28.617506 (+ 71us) client_negotiation.cc:240] Negotiation successful
0131 03:08:28.617604 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260131 03:08:28.617728 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.611596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34660 (local address 127.19.84.61:38999)
0131 03:08:28.611631 (+ 35us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:28.611633 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:28.611893 (+ 260us) server_negotiation.cc:408] Connection header received
0131 03:08:28.611951 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:28.611953 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:28.611965 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:28.612021 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:28.612947 (+ 926us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.613484 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.614053 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.614435 (+ 382us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.615359 (+ 924us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:28.615362 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:28.615362 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:28.615368 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:28.616733 (+ 1365us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.617052 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.617053 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.617053 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.617080 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.617226 (+ 146us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.617227 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.617227 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.617335 (+ 108us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:28.617411 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:28.617525 (+ 114us) server_negotiation.cc:300] Negotiation successful
0131 03:08:28.617653 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":16}
I20260131 03:08:28.642478 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.611626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58386 (local address 127.19.84.62:33805)
0131 03:08:28.611666 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:28.611681 (+ 15us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:28.611927 (+ 246us) server_negotiation.cc:408] Connection header received
0131 03:08:28.612003 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:28.612004 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:28.612014 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:28.612070 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:28.612584 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.613191 (+ 607us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.613966 (+ 775us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.614154 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.639923 (+ 25769us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:28.639930 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:28.639931 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:28.639940 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:28.641431 (+ 1491us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.641811 (+ 380us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.641812 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.641813 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.641834 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.641999 (+ 165us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.642000 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.642000 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.642108 (+ 108us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:28.642165 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:28.642336 (+ 171us) server_negotiation.cc:300] Negotiation successful
0131 03:08:28.642408 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260131 03:08:28.642546 27113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.611731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.0.0.1:58386)
0131 03:08:28.611864 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:28.611874 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:28.611949 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:28.612073 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:28.612074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:28.612327 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:28.612494 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.612495 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.613289 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.613290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:28.613876 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.613879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.613956 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.614366 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:28.614371 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:28.639781 (+ 25410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:28.641518 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.641519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.641520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.641724 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.641893 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.641894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.641894 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.641926 (+ 32us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.642173 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:28.642174 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:28.642234 (+ 60us) client_negotiation.cc:769] Sending connection context
0131 03:08:28.642309 (+ 75us) client_negotiation.cc:240] Negotiation successful
0131 03:08:28.642427 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
Jan 31 03:08:28 dist-test-slave-5nnp krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828907, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.62@KRBTEST.COM for oryx/127.19.84.61@KRBTEST.COM
I20260131 03:08:28.651073 27114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.644647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:38999 (local address 127.0.0.1:34670)
0131 03:08:28.644806 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:28.644838 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:28.644930 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:28.645091 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:28.645095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:28.645400 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:28.645587 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.645589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.646184 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.646185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:28.646531 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.646532 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.646620 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.647077 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:28.647081 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:28.648408 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:28.649974 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.649980 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.649984 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.650226 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.650413 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.650414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.650414 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.650443 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.650719 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:28.650724 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:28.650783 (+ 59us) client_negotiation.cc:769] Sending connection context
0131 03:08:28.650874 (+ 91us) client_negotiation.cc:240] Negotiation successful
0131 03:08:28.650971 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20260131 03:08:28.651135 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.644703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34670 (local address 127.19.84.61:38999)
0131 03:08:28.644740 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:28.644742 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:28.644902 (+ 160us) server_negotiation.cc:408] Connection header received
0131 03:08:28.644996 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:28.644997 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:28.645009 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:28.645086 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:28.645674 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.646105 (+ 431us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.646613 (+ 508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.646789 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.648495 (+ 1706us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:28.648499 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:28.648499 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:28.648506 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:28.649862 (+ 1356us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.650319 (+ 457us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.650320 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.650321 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.650344 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.650513 (+ 169us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.650514 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.650515 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.650635 (+ 120us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:28.650704 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:28.650916 (+ 212us) server_negotiation.cc:300] Negotiation successful
0131 03:08:28.651003 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260131 03:08:28.652657 26541 catalog_manager.cc:7141] Initiating ChangeConfig request to add master 127.19.84.61:38999: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403"
member_type: NON_VOTER
last_known_addr {
host: "127.19.84.61"
port: 38999
}
attrs {
promote: true
}
}
dest_uuid: "6c41b3e9bc84467e9d9758cbd2691e72"
cas_config_opid_index: -1
I20260131 03:08:28.652956 26541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: NON_VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: true } }
I20260131 03:08:28.653566 27116 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b75c66249f3f4d7d8a97151e41f9f403 (127.19.84.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: NON_VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: true } } }
I20260131 03:08:28.653666 27062 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:28.654712 27116 catalog_manager.cc:7145] Successfully completed master ChangeConfig request to add master 127.19.84.61:38999
I20260131 03:08:28.654772 27062 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 1 FOLLOWER]: Refusing update from remote peer 6c41b3e9bc84467e9d9758cbd2691e72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260131 03:08:28.655085 27020 master.cc:561] Master@127.19.84.61:38999 shutting down...
I20260131 03:08:28.655264 27116 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: NON_VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:08:28.655449 27117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: NON_VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: true } } }
I20260131 03:08:28.655546 27117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [sys.catalog]: This master's current role is: LEADER
W20260131 03:08:28.656842 26535 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260131 03:08:28.658465 26535 consensus_peers.cc:597] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 -> Peer b75c66249f3f4d7d8a97151e41f9f403 (127.19.84.61:38999): Couldn't send request to peer b75c66249f3f4d7d8a97151e41f9f403. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260131 03:08:28.658488 27020 raft_consensus.cc:2243] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:08:28.658566 27020 raft_consensus.cc:2272] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:08:28.658607 27020 tablet_replica.cc:333] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403: stopping tablet replica
WARNING: no policy specified for oryx/127.19.84.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.84.60@KRBTEST.COM" created.
I20260131 03:08:28.671175 27020 master.cc:583] Master@127.19.84.61:38999 shutdown complete.
I20260131 03:08:28.671236 27020 master_runner.cc:305] Clearing existing system tablet
I20260131 03:08:28.671393 27020 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:28.671439 27020 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:28.673131 27020 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:08:28.673763 27123 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:28.673933 27020 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:28.674001 27020 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:08:28.674031 27020 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/wal
uuid: "b75c66249f3f4d7d8a97151e41f9f403"
format_stamp: "Formatted at 2026-01-31 03:08:28 on dist-test-slave-5nnp"
server_key: "64d594dd0acf749838672ee8aa8137f0"
server_key_iv: "88fe832738e828c7e30c0b3c5f193bd9"
server_key_version: "encryptionkey@0"
I20260131 03:08:28.675201 27020 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.84.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.19.84.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab.
I20260131 03:08:28.676637 27020 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260131 03:08:28.676707 27020 log.cc:1199] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260131 03:08:28.676903 27020 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403: Deleting consensus metadata
I20260131 03:08:28.677101 27020 master_runner.cc:315] Copying system tablet from 127.19.84.62:33805
I20260131 03:08:28.677742 27020 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403: tablet copy: Beginning tablet copy session from remote peer at address 127.19.84.62:33805
WARNING: no policy specified for HTTP/127.19.84.60@KRBTEST.COM; defaulting to no policy
I20260131 03:08:28.684621 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.678148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58396 (local address 127.19.84.62:33805)
0131 03:08:28.678194 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:28.678197 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:28.678269 (+ 72us) server_negotiation.cc:408] Connection header received
0131 03:08:28.678339 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:28.678340 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:28.678352 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:28.678431 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:28.678974 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.679838 (+ 864us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.680615 (+ 777us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.680848 (+ 233us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.681712 (+ 864us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:28.681717 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:28.681718 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:28.681734 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:28.683490 (+ 1756us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.683864 (+ 374us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.683865 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.683866 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.683895 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.684103 (+ 208us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.684104 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.684104 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.684227 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:28.684297 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:28.684481 (+ 184us) server_negotiation.cc:300] Negotiation successful
0131 03:08:28.684552 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260131 03:08:28.684677 27130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.678066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.0.0.1:58396)
0131 03:08:28.678209 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:28.678220 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:28.678282 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:28.678453 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:28.678454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:28.678636 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:28.678869 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.678871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.679931 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.679932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:28.680515 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.680517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.680599 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.681247 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:28.681253 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:28.681615 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:28.683577 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.683579 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.683579 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.683782 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.683987 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.683988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.683988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.684030 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.684327 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:28.684328 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:28.684376 (+ 48us) client_negotiation.cc:769] Sending connection context
0131 03:08:28.684449 (+ 73us) client_negotiation.cc:240] Negotiation successful
0131 03:08:28.684562 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
I20260131 03:08:28.684962 26567 tablet_copy_service.cc:140] P 6c41b3e9bc84467e9d9758cbd2691e72: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b75c66249f3f4d7d8a97151e41f9f403 ({username='oryx', principal='oryx/127.19.84.61@KRBTEST.COM'} at 127.0.0.1:58396)
I20260131 03:08:28.685056 26567 tablet_copy_service.cc:161] P 6c41b3e9bc84467e9d9758cbd2691e72: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b75c66249f3f4d7d8a97151e41f9f403 at {username='oryx', principal='oryx/127.19.84.61@KRBTEST.COM'} at 127.0.0.1:58396: session id = b75c66249f3f4d7d8a97151e41f9f403-00000000000000000000000000000000
I20260131 03:08:28.686477 26567 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72: Tablet Copy: opened 0 blocks and 1 log segments
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.60@KRBTEST.COM" created.
I20260131 03:08:28.687150 27020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260131 03:08:28.688449 27020 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403: tablet copy: Starting download of 0 data blocks...
I20260131 03:08:28.688593 27020 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403: tablet copy: Starting download of 1 WAL segments...
I20260131 03:08:28.689520 27020 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260131 03:08:28.690459 26567 tablet_copy_service.cc:342] P 6c41b3e9bc84467e9d9758cbd2691e72: Request end of tablet copy session b75c66249f3f4d7d8a97151e41f9f403-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.84.61@KRBTEST.COM'} at 127.0.0.1:58396
I20260131 03:08:28.690532 26567 tablet_copy_service.cc:434] P 6c41b3e9bc84467e9d9758cbd2691e72: ending tablet copy session b75c66249f3f4d7d8a97151e41f9f403-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b75c66249f3f4d7d8a97151e41f9f403
W20260131 03:08:28.692101 27020 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260131 03:08:28.693055 27020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:28.694290 27132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:28.694299 27135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:28.694486 27020 server_base.cc:1060] running on GCE node
W20260131 03:08:28.696311 27133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:28.696521 27020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:28.696676 27020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:28.697805 27020 hybrid_clock.cc:648] HybridClock initialized: now 1769828908697780 us; error 46 us; skew 500 ppm
Jan 31 03:08:28 dist-test-slave-5nnp krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828908, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.19.84.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab.
I20260131 03:08:28.702329 27020 init.cc:377] Logged in from keytab as oryx/127.19.84.61@KRBTEST.COM (short username oryx)
I20260131 03:08:28.702661 27020 webserver.cc:492] Webserver started at http://127.19.84.61:33955/ using document root <none> and password file <none>
I20260131 03:08:28.702790 27020 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:28.702858 27020 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:28.703050 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.60:36003
--webserver_interface=127.19.84.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:37697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:33805,127.19.84.61:38999,127.19.84.60:36003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260131 03:08:28.703629 27020 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:08:28.704181 27142 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:28.704370 27020 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260131 03:08:28.704423 27020 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/wal
uuid: "b75c66249f3f4d7d8a97151e41f9f403"
format_stamp: "Formatted at 2026-01-31 03:08:28 on dist-test-slave-5nnp"
server_key: "64d594dd0acf749838672ee8aa8137f0"
server_key_iv: "88fe832738e828c7e30c0b3c5f193bd9"
server_key_version: "encryptionkey@0"
I20260131 03:08:28.704495 27020 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:28.727701 27020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:28.729528 27020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:28.732647 27020 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.61:38999
I20260131 03:08:28.732743 27195 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.61:38999 every 8 connection(s)
I20260131 03:08:28.732973 27020 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-1/data/info.pb
I20260131 03:08:28.736092 27196 sys_catalog.cc:263] Verifying existing consensus state
I20260131 03:08:28.736593 27196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403: Bootstrap starting.
I20260131 03:08:28.739228 27196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260131 03:08:28.739465 27196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403: Bootstrap complete.
I20260131 03:08:28.739688 27196 raft_consensus.cc:359] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: NON_VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: true } }
I20260131 03:08:28.739766 27196 raft_consensus.cc:740] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b75c66249f3f4d7d8a97151e41f9f403, State: Initialized, Role: LEARNER
I20260131 03:08:28.739800 27196 consensus_queue.cc:260] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: NON_VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: true } }
I20260131 03:08:28.739971 27196 sys_catalog.cc:565] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:08:28.739969 27199 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: NON_VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: true } } }
I20260131 03:08:28.740187 27199 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [sys.catalog]: This master's current role is: LEARNER
I20260131 03:08:28.741600 27210 catalog_manager.cc:1269] Loaded cluster ID: 00b8836497954b1f8f0752703e43dd26
I20260131 03:08:28.741698 27210 catalog_manager.cc:1562] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403: loading cluster ID for follower catalog manager: success
I20260131 03:08:28.743096 27210 catalog_manager.cc:1584] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403: acquiring CA information for follower catalog manager: success
I20260131 03:08:28.743415 27210 catalog_manager.cc:1612] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260131 03:08:28.793625 27143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:28.793838 27143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:28.793880 27143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:28.795308 27143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:08:28.795377 27143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:28.795430 27143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:28.795460 27143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:08:28.795485 27143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:08:28.797098 27143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:37697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:33805,127.19.84.61:38999,127.19.84.60:36003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.60:36003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.84.60
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:28.797364 27143 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:28.797567 27143 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:28.800104 27218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:28.800101 27215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:28.800109 27216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:28.800323 27143 server_base.cc:1060] running on GCE node
I20260131 03:08:28.800561 27143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:28.800812 27143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:28.801962 27143 hybrid_clock.cc:648] HybridClock initialized: now 1769828908801955 us; error 30 us; skew 500 ppm
Jan 31 03:08:28 dist-test-slave-5nnp krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828908, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:28.804314 27143 init.cc:377] Logged in from keytab as oryx/127.19.84.60@KRBTEST.COM (short username oryx)
I20260131 03:08:28.804957 27143 webserver.cc:492] Webserver started at http://127.19.84.60:37765/ using document root <none> and password file <none>
I20260131 03:08:28.805414 27143 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:28.805459 27143 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:28.805573 27143 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:28.806553 27143 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/data/instance:
uuid: "005eb390434c4f61a808cc442be89188"
format_stamp: "Formatted at 2026-01-31 03:08:28 on dist-test-slave-5nnp"
server_key: "63a3a07a0faeb80786f398020cbb572e"
server_key_iv: "22ddcd0ee55ca85531a2391dcd8dca9a"
server_key_version: "encryptionkey@0"
I20260131 03:08:28.806903 27143 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/wal/instance:
uuid: "005eb390434c4f61a808cc442be89188"
format_stamp: "Formatted at 2026-01-31 03:08:28 on dist-test-slave-5nnp"
server_key: "63a3a07a0faeb80786f398020cbb572e"
server_key_iv: "22ddcd0ee55ca85531a2391dcd8dca9a"
server_key_version: "encryptionkey@0"
I20260131 03:08:28.808375 27143 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:08:28.809144 27225 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:28.809440 27143 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:28.809523 27143 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/wal
uuid: "005eb390434c4f61a808cc442be89188"
format_stamp: "Formatted at 2026-01-31 03:08:28 on dist-test-slave-5nnp"
server_key: "63a3a07a0faeb80786f398020cbb572e"
server_key_iv: "22ddcd0ee55ca85531a2391dcd8dca9a"
server_key_version: "encryptionkey@0"
I20260131 03:08:28.809573 27143 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:28.830000 27143 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:28.831801 27143 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:28.831910 27143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:28.835839 27143 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.60:36003
I20260131 03:08:28.835867 27277 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.60:36003 every 8 connection(s)
I20260131 03:08:28.836182 27143 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/data/info.pb
I20260131 03:08:28.836819 27278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:28.838387 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 27143
I20260131 03:08:28.839161 27278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } has no permanent_uuid. Determining permanent_uuid...
Jan 31 03:08:28 dist-test-slave-5nnp krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828908, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.60@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM
Jan 31 03:08:28 dist-test-slave-5nnp krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.84.60@KRBTEST.COM
I20260131 03:08:28.847591 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.840139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58406 (local address 127.19.84.62:33805)
0131 03:08:28.840189 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:28.840192 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:28.840232 (+ 40us) server_negotiation.cc:408] Connection header received
0131 03:08:28.840287 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:28.840288 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:28.840299 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:28.840368 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:28.841128 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.841719 (+ 591us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.842256 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.842438 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.844556 (+ 2118us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:28.844561 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:28.844562 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:28.844569 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:28.846170 (+ 1601us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.846659 (+ 489us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.846660 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.846661 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.846682 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.846890 (+ 208us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.846890 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.846890 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.846976 (+ 86us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:28.847026 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:28.847336 (+ 310us) server_negotiation.cc:300] Negotiation successful
0131 03:08:28.847478 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260131 03:08:28.847627 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.840004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.0.0.1:58406)
0131 03:08:28.840151 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:28.840180 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:28.840241 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:28.840386 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:28.840387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:28.840704 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:28.841041 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.841045 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.841797 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.841799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:28.842181 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.842183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.842266 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.842844 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:28.842865 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:28.844469 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:28.846252 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.846259 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.846262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.846584 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.846746 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.846747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.846748 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.846832 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.847037 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:28.847042 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:28.847188 (+ 146us) client_negotiation.cc:769] Sending connection context
0131 03:08:28.847318 (+ 130us) client_negotiation.cc:240] Negotiation successful
0131 03:08:28.847489 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
I20260131 03:08:28.848217 27278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } has no permanent_uuid. Determining permanent_uuid...
I20260131 03:08:28.848636 27280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.839030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42224 (local address 127.19.84.60:36003)
0131 03:08:28.839195 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:28.839204 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:28.839223 (+ 19us) server_negotiation.cc:408] Connection header received
0131 03:08:28.839598 (+ 375us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:28.839604 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:28.839670 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:28.839802 (+ 132us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:28.840687 (+ 885us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.841783 (+ 1096us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.842343 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.842550 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.845241 (+ 2691us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:28.845250 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:28.845250 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:28.845257 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:28.847321 (+ 2064us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.847715 (+ 394us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.847718 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.847719 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.847761 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.848063 (+ 302us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.848064 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.848064 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.848194 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:28.848327 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:28.848460 (+ 133us) server_negotiation.cc:300] Negotiation successful
0131 03:08:28.848546 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
Jan 31 03:08:28 dist-test-slave-5nnp krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828908, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.60@KRBTEST.COM for oryx/127.19.84.61@KRBTEST.COM
I20260131 03:08:28.854218 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.848474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:38999 (local address 127.0.0.1:34678)
0131 03:08:28.848674 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:28.848837 (+ 163us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:28.848889 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:28.849039 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:28.849040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:28.849205 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:28.849357 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.849358 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.850166 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.850167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:28.850512 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.850513 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.850572 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.850973 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:28.850977 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:28.852109 (+ 1132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:28.853559 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.853560 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.853560 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.853686 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.853826 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.853827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.853828 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.853848 (+ 20us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.854010 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:28.854010 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:28.854040 (+ 30us) client_negotiation.cc:769] Sending connection context
0131 03:08:28.854088 (+ 48us) client_negotiation.cc:240] Negotiation successful
0131 03:08:28.854157 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260131 03:08:28.854231 27282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.848584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34678 (local address 127.19.84.61:38999)
0131 03:08:28.848699 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:28.848701 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:28.848895 (+ 194us) server_negotiation.cc:408] Connection header received
0131 03:08:28.848945 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:28.848946 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:28.848957 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:28.849020 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:28.849425 (+ 405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.850098 (+ 673us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.850578 (+ 480us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.850688 (+ 110us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.852187 (+ 1499us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:28.852191 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:28.852191 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:28.852196 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:28.853483 (+ 1287us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.853746 (+ 263us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.853747 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.853747 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.853774 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.853890 (+ 116us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.853890 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.853890 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.853964 (+ 74us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:28.854008 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:28.854101 (+ 93us) server_negotiation.cc:300] Negotiation successful
0131 03:08:28.854166 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":64,"threads_started":1}
I20260131 03:08:28.854658 27278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } has no permanent_uuid. Determining permanent_uuid...
Jan 31 03:08:28 dist-test-slave-5nnp krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828908, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.60@KRBTEST.COM for oryx/127.19.84.60@KRBTEST.COM
I20260131 03:08:28.859859 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.854911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:36003 (local address 127.0.0.1:42238)
0131 03:08:28.854957 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:28.854967 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:28.855024 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:28.855140 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:28.855141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:28.855265 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:28.855414 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.855415 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.856010 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.856011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:28.856315 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.856316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.856403 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.856745 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:28.856748 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:28.857751 (+ 1003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:28.859115 (+ 1364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.859116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.859117 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.859242 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.859436 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.859436 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.859437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.859460 (+ 23us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.859638 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:28.859638 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:28.859670 (+ 32us) client_negotiation.cc:769] Sending connection context
0131 03:08:28.859725 (+ 55us) client_negotiation.cc:240] Negotiation successful
0131 03:08:28.859801 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260131 03:08:28.859927 27280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.854973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42238 (local address 127.19.84.60:36003)
0131 03:08:28.855026 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:28.855028 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:28.855036 (+ 8us) server_negotiation.cc:408] Connection header received
0131 03:08:28.855075 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:28.855076 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:28.855084 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:28.855137 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:28.855493 (+ 356us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.855933 (+ 440us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.856401 (+ 468us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.856590 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.857831 (+ 1241us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:28.857834 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:28.857834 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:28.857840 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:28.859045 (+ 1205us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.859304 (+ 259us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.859305 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.859305 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.859322 (+ 17us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.859500 (+ 178us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.859501 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.859501 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.859586 (+ 85us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:28.859634 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:28.859741 (+ 107us) server_negotiation.cc:300] Negotiation successful
0131 03:08:28.859848 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":38}
I20260131 03:08:28.861181 27278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188: Bootstrap starting.
I20260131 03:08:28.861979 27278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:28.862239 27278 log.cc:826] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:28.862955 27278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188: No bootstrap required, opened a new log
I20260131 03:08:28.864111 27278 raft_consensus.cc:359] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } }
I20260131 03:08:28.864240 27278 raft_consensus.cc:385] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:28.864284 27278 raft_consensus.cc:740] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 005eb390434c4f61a808cc442be89188, State: Initialized, Role: FOLLOWER
I20260131 03:08:28.864388 27278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } }
I20260131 03:08:28.864642 27284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } } }
I20260131 03:08:28.864655 27278 sys_catalog.cc:565] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:08:28.864753 27284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [sys.catalog]: This master's current role is: FOLLOWER
W20260131 03:08:28.866425 27295 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:08:28.866573 27295 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:08:28.868507 27143 master_runner.cc:428] Detected that this master 005eb390434c4f61a808cc442be89188 is joining an existing cluster
I20260131 03:08:28.868552 27143 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.84.60:36003
I20260131 03:08:28.898289 27280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.869993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42254 (local address 127.19.84.60:36003)
0131 03:08:28.870027 (+ 34us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:28.870028 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:28.870139 (+ 111us) server_negotiation.cc:408] Connection header received
0131 03:08:28.870169 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:28.870170 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:28.870177 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:28.870219 (+ 42us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:28.892330 (+ 22111us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.893019 (+ 689us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.893752 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.893923 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.895242 (+ 1319us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:28.895245 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:28.895246 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:28.895252 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:28.897164 (+ 1912us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.897511 (+ 347us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.897512 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.897513 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.897532 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.897699 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.897700 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.897701 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.897807 (+ 106us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:28.897851 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:28.898025 (+ 174us) server_negotiation.cc:300] Negotiation successful
0131 03:08:28.898160 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260131 03:08:28.898373 27282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.869478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34688 (local address 127.19.84.61:38999)
0131 03:08:28.869523 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:28.869526 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:28.869809 (+ 283us) server_negotiation.cc:408] Connection header received
0131 03:08:28.869896 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:28.869897 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:28.869911 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:28.869961 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:28.892380 (+ 22419us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.893123 (+ 743us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.893850 (+ 727us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.894014 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.895632 (+ 1618us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:28.895636 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:28.895636 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:28.895644 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:28.897328 (+ 1684us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.897671 (+ 343us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.897672 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.897672 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.897695 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.897862 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.897863 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.897864 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.897962 (+ 98us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:28.898004 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:28.898189 (+ 185us) server_negotiation.cc:300] Negotiation successful
0131 03:08:28.898272 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260131 03:08:28.898504 27301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.869622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:36003 (local address 127.0.0.1:42254)
0131 03:08:28.870066 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:28.870075 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:28.870132 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:28.870223 (+ 91us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:28.870224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:28.891855 (+ 21631us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:28.892201 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.892204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.893115 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.893116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:28.893665 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.893667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.893976 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.894569 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:28.894574 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:28.895150 (+ 576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:28.897254 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.897255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.897256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.897436 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.897609 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.897610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.897610 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.897639 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.897890 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:28.897891 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:28.897923 (+ 32us) client_negotiation.cc:769] Sending connection context
0131 03:08:28.898138 (+ 215us) client_negotiation.cc:240] Negotiation successful
0131 03:08:28.898215 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":65,"threads_started":1}
I20260131 03:08:28.899024 27303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.869526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:38999 (local address 127.0.0.1:34688)
0131 03:08:28.869740 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:28.869749 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:28.869831 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:28.870011 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:28.870011 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:28.891855 (+ 21844us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:28.892201 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.892204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.893212 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.893213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:28.893766 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.893767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.894056 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.894933 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:28.894936 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:28.895523 (+ 587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:28.897421 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.897422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.897422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.897593 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.897761 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.897762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.897762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.897784 (+ 22us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.898048 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:28.898049 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:28.898079 (+ 30us) client_negotiation.cc:769] Sending connection context
0131 03:08:28.898761 (+ 682us) client_negotiation.cc:240] Negotiation successful
0131 03:08:28.898910 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":75,"threads_started":1}
I20260131 03:08:28.899606 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.869736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58416 (local address 127.19.84.62:33805)
0131 03:08:28.869821 (+ 85us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:28.869823 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:28.869832 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:08:28.869855 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:28.869856 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:28.869867 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:28.869922 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:28.892331 (+ 22409us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.893121 (+ 790us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.893936 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.894118 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.895361 (+ 1243us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:28.895365 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:28.895366 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:28.895393 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:28.897905 (+ 2512us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.898768 (+ 863us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.898769 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.898769 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.898796 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.898959 (+ 163us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.898960 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.898960 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.899062 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:28.899100 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:28.899435 (+ 335us) server_negotiation.cc:300] Negotiation successful
0131 03:08:28.899516 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260131 03:08:28.900022 27302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.869405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.0.0.1:58416)
0131 03:08:28.869655 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:28.869665 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:28.869740 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:28.869933 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:28.869934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:28.891854 (+ 21920us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:28.892169 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.892175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.893225 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.893226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:28.893833 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.893835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.894184 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.894733 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:28.894737 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:28.895266 (+ 529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:28.897994 (+ 2728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.897995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.897996 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.898304 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.898866 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.898866 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.898867 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.898897 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.899254 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:28.899255 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:28.899295 (+ 40us) client_negotiation.cc:769] Sending connection context
0131 03:08:28.899824 (+ 529us) client_negotiation.cc:240] Negotiation successful
0131 03:08:28.899922 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":82,"threads_started":1}
I20260131 03:08:28.903251 27114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.900377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.0.0.1:58418)
0131 03:08:28.900419 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:28.900431 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:28.900476 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:28.900977 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:28.900977 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:28.900979 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:28.901166 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.901167 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.902036 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.902036 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:28.902973 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.902975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.903049 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.903093 (+ 44us) client_negotiation.cc:769] Sending connection context
0131 03:08:28.903128 (+ 35us) client_negotiation.cc:240] Negotiation successful
0131 03:08:28.903167 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260131 03:08:28.903683 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.900480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58418 (local address 127.19.84.62:33805)
0131 03:08:28.900835 (+ 355us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:28.900836 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:28.900845 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:08:28.900879 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:28.900880 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:28.900889 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:28.900941 (+ 52us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:28.901230 (+ 289us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.901956 (+ 726us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.903057 (+ 1101us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.903530 (+ 473us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.903558 (+ 28us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:28.903580 (+ 22us) server_negotiation.cc:300] Negotiation successful
0131 03:08:28.903608 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334}
Jan 31 03:08:28 dist-test-slave-5nnp krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828907, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.62@KRBTEST.COM for oryx/127.19.84.60@KRBTEST.COM
I20260131 03:08:28.907760 27114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.904693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:38999 (local address 127.0.0.1:34690)
0131 03:08:28.904734 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:28.904743 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:28.904778 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:28.904961 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:28.904962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:28.904963 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:28.905139 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.905141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.906319 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.906320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:28.907544 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.907547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.907623 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.907627 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:08:28.907660 (+ 33us) client_negotiation.cc:240] Negotiation successful
0131 03:08:28.907699 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260131 03:08:28.908317 27282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.904742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34690 (local address 127.19.84.61:38999)
0131 03:08:28.904852 (+ 110us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:28.904854 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:28.904860 (+ 6us) server_negotiation.cc:408] Connection header received
0131 03:08:28.904876 (+ 16us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:28.904877 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:28.904892 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:28.904958 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:28.905219 (+ 261us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.906223 (+ 1004us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.907634 (+ 1411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.908075 (+ 441us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.908167 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:28.908189 (+ 22us) server_negotiation.cc:300] Negotiation successful
0131 03:08:28.908224 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260131 03:08:28.908466 27304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.901801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:36003 (local address 127.0.0.1:42260)
0131 03:08:28.901959 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:28.901968 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:28.902028 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:28.902166 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:28.902167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:28.902328 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:28.902464 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.902465 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.903106 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.903107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:28.903600 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.903602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.903675 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.904073 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:28.904079 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:28.905597 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:28.907517 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.907518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.907518 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.907705 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.907901 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.907901 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.907902 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.907929 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.908160 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:28.908161 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:28.908202 (+ 41us) client_negotiation.cc:769] Sending connection context
0131 03:08:28.908276 (+ 74us) client_negotiation.cc:240] Negotiation successful
0131 03:08:28.908371 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":119,"threads_started":1}
I20260131 03:08:28.908480 27280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.901841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42260 (local address 127.19.84.60:36003)
0131 03:08:28.901877 (+ 36us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:28.901880 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:28.902031 (+ 151us) server_negotiation.cc:408] Connection header received
0131 03:08:28.902098 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:28.902099 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:28.902108 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:28.902162 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:28.902554 (+ 392us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.903009 (+ 455us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.903688 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.903862 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.905691 (+ 1829us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:28.905694 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:28.905694 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:28.905700 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:28.907436 (+ 1736us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.907810 (+ 374us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.907811 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.907811 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.907834 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.907983 (+ 149us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.907984 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.907984 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.908083 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:28.908153 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:28.908315 (+ 162us) server_negotiation.cc:300] Negotiation successful
0131 03:08:28.908394 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260131 03:08:28.909032 26540 catalog_manager.cc:7141] Initiating ChangeConfig request to add master 127.19.84.60:36003: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "005eb390434c4f61a808cc442be89188"
member_type: NON_VOTER
last_known_addr {
host: "127.19.84.60"
port: 36003
}
attrs {
promote: true
}
}
dest_uuid: "6c41b3e9bc84467e9d9758cbd2691e72"
cas_config_opid_index: 5
I20260131 03:08:28.909252 26540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: NON_VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: true } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: NON_VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: true } }
I20260131 03:08:28.909600 27116 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 005eb390434c4f61a808cc442be89188 (127.19.84.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: NON_VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: true } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: NON_VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: true } } }
I20260131 03:08:28.909793 27170 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 1 LEARNER]: Refusing update from remote peer 6c41b3e9bc84467e9d9758cbd2691e72: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260131 03:08:28.909866 27252 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:28.909996 27118 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: NON_VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260131 03:08:28.910570 27116 catalog_manager.cc:7145] Successfully completed master ChangeConfig request to add master 127.19.84.60:36003
I20260131 03:08:28.910497 27170 raft_consensus.cc:2955] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 005eb390434c4f61a808cc442be89188 (127.19.84.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: NON_VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: true } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: NON_VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: true } } }
I20260131 03:08:28.910648 27116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: NON_VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: true } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: NON_VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: true } } }
I20260131 03:08:28.910717 27116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:28.910781 27252 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 1 FOLLOWER]: Refusing update from remote peer 6c41b3e9bc84467e9d9758cbd2691e72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260131 03:08:28.910993 27143 master.cc:561] Master@127.19.84.60:36003 shutting down...
I20260131 03:08:28.911593 27306 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c41b3e9bc84467e9d9758cbd2691e72. Latest consensus state: current_term: 1 leader_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: NON_VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: true } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: NON_VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: true } } }
I20260131 03:08:28.911620 27199 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: NON_VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: true } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: NON_VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: true } } }
I20260131 03:08:28.911669 27306 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [sys.catalog]: This master's current role is: LEARNER
I20260131 03:08:28.911703 27199 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [sys.catalog]: This master's current role is: LEARNER
I20260131 03:08:28.912091 27116 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72: attempting to promote NON_VOTER b75c66249f3f4d7d8a97151e41f9f403 to VOTER
I20260131 03:08:28.912122 27117 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: NON_VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:08:28.912343 27116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: NON_VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: true } }
W20260131 03:08:28.913800 26533 consensus_peers.cc:597] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 -> Peer 005eb390434c4f61a808cc442be89188 (127.19.84.60:36003): Couldn't send request to peer 005eb390434c4f61a808cc442be89188. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
W20260131 03:08:28.913962 26540 master.cc:679] Remote error: unable to get registration information for peer 005eb390434c4f61a808cc442be89188: Service unavailable: service kudu.master.MasterService not registered on Master
I20260131 03:08:28.914139 27143 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:08:28.914225 27143 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:08:28.914266 27143 tablet_replica.cc:333] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188: stopping tablet replica
I20260131 03:08:28.915534 27170 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 1 LEARNER]: Refusing update from remote peer 6c41b3e9bc84467e9d9758cbd2691e72: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260131 03:08:28.915735 27118 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260131 03:08:28.916463 27116 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b75c66249f3f4d7d8a97151e41f9f403 (127.19.84.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: NON_VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: true } } }
I20260131 03:08:28.916627 27170 raft_consensus.cc:2955] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b75c66249f3f4d7d8a97151e41f9f403 (127.19.84.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: NON_VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: true } } }
I20260131 03:08:28.917318 27116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: NON_VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: true } } }
I20260131 03:08:28.917371 27118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: NON_VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: true } } }
I20260131 03:08:28.917403 27116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:28.917443 27118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:28.917559 27199 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: NON_VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: true } } }
I20260131 03:08:28.917647 27199 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:08:28.918915 27304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.918775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:36003 (local address 127.0.0.1:42264)
0131 03:08:28.918825 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:28.918880 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:36003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260131 03:08:28.919032 26540 master.cc:679] Network error: unable to get registration information for peer 005eb390434c4f61a808cc442be89188: Client connection negotiation failed: client connection to 127.19.84.60:36003: connect: Connection refused (error 111)
I20260131 03:08:28.924928 27304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.924799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:36003 (local address 127.0.0.1:42274)
0131 03:08:28.924853 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:28.924890 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:36003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260131 03:08:28.925074 26540 master.cc:679] Network error: unable to get registration information for peer 005eb390434c4f61a808cc442be89188: Client connection negotiation failed: client connection to 127.19.84.60:36003: connect: Connection refused (error 111)
I20260131 03:08:28.926808 27143 master.cc:583] Master@127.19.84.60:36003 shutdown complete.
I20260131 03:08:28.926877 27143 master_runner.cc:305] Clearing existing system tablet
I20260131 03:08:28.927042 27143 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:28.927103 27143 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:28.927783 27143 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260131 03:08:28.928287 27308 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:28.928473 27143 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:28.928545 27143 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260131 03:08:28.928575 27143 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/wal
uuid: "005eb390434c4f61a808cc442be89188"
format_stamp: "Formatted at 2026-01-31 03:08:28 on dist-test-slave-5nnp"
server_key: "63a3a07a0faeb80786f398020cbb572e"
server_key_iv: "22ddcd0ee55ca85531a2391dcd8dca9a"
server_key_version: "encryptionkey@0"
I20260131 03:08:28.929318 27143 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:08:28.930763 27143 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260131 03:08:28.930833 27143 log.cc:1199] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260131 03:08:28.931021 27143 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188: Deleting consensus metadata
I20260131 03:08:28.931141 27143 master_runner.cc:315] Copying system tablet from 127.19.84.62:33805
I20260131 03:08:28.931841 27143 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188: tablet copy: Beginning tablet copy session from remote peer at address 127.19.84.62:33805
I20260131 03:08:28.935138 27304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.935014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:36003 (local address 127.0.0.1:42278)
0131 03:08:28.935064 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:28.935098 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:36003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260131 03:08:28.935284 26540 master.cc:679] Network error: unable to get registration information for peer 005eb390434c4f61a808cc442be89188: Client connection negotiation failed: client connection to 127.19.84.60:36003: connect: Connection refused (error 111)
I20260131 03:08:28.938216 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.932222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58434 (local address 127.19.84.62:33805)
0131 03:08:28.932286 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:28.932289 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:28.932352 (+ 63us) server_negotiation.cc:408] Connection header received
0131 03:08:28.932426 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:28.932428 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:28.932439 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:28.932494 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:28.932968 (+ 474us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.933744 (+ 776us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.934548 (+ 804us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.934693 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.935747 (+ 1054us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:28.935750 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:28.935751 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:28.935757 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:28.937213 (+ 1456us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.937577 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.937578 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.937579 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.937602 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.937761 (+ 159us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.937762 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.937762 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.937872 (+ 110us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:28.937933 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:28.938089 (+ 156us) server_negotiation.cc:300] Negotiation successful
0131 03:08:28.938153 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260131 03:08:28.938283 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.932122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.0.0.1:58434)
0131 03:08:28.932286 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:28.932297 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:28.932363 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:28.932509 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:28.932510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:28.932705 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:28.932891 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.932893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.933855 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.933856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:28.934462 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.934464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.934541 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.935274 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:28.935280 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:28.935669 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:28.937298 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.937300 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.937300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.937496 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.937669 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.937670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.937670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.937704 (+ 34us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.937938 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:28.937939 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:28.937989 (+ 50us) client_negotiation.cc:769] Sending connection context
0131 03:08:28.938066 (+ 77us) client_negotiation.cc:240] Negotiation successful
0131 03:08:28.938175 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
I20260131 03:08:28.938504 26567 tablet_copy_service.cc:140] P 6c41b3e9bc84467e9d9758cbd2691e72: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 005eb390434c4f61a808cc442be89188 ({username='oryx', principal='oryx/127.19.84.60@KRBTEST.COM'} at 127.0.0.1:58434)
I20260131 03:08:28.938558 26567 tablet_copy_service.cc:161] P 6c41b3e9bc84467e9d9758cbd2691e72: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 005eb390434c4f61a808cc442be89188 at {username='oryx', principal='oryx/127.19.84.60@KRBTEST.COM'} at 127.0.0.1:58434: session id = 005eb390434c4f61a808cc442be89188-00000000000000000000000000000000
I20260131 03:08:28.939038 26567 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72: Tablet Copy: opened 0 blocks and 1 log segments
I20260131 03:08:28.939677 27143 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260131 03:08:28.940614 27143 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188: tablet copy: Starting download of 0 data blocks...
I20260131 03:08:28.940764 27143 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188: tablet copy: Starting download of 1 WAL segments...
I20260131 03:08:28.941651 27143 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260131 03:08:28.942567 26567 tablet_copy_service.cc:342] P 6c41b3e9bc84467e9d9758cbd2691e72: Request end of tablet copy session 005eb390434c4f61a808cc442be89188-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.84.60@KRBTEST.COM'} at 127.0.0.1:58434
I20260131 03:08:28.942636 26567 tablet_copy_service.cc:434] P 6c41b3e9bc84467e9d9758cbd2691e72: ending tablet copy session 005eb390434c4f61a808cc442be89188-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 005eb390434c4f61a808cc442be89188
W20260131 03:08:28.944751 27143 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260131 03:08:28.945595 27143 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:28.947060 27316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:28.947113 27315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:28.947206 27143 server_base.cc:1060] running on GCE node
W20260131 03:08:28.947513 27318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:28.947690 27143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:28.947883 27143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:28.949023 27143 hybrid_clock.cc:648] HybridClock initialized: now 1769828908949017 us; error 25 us; skew 500 ppm
Jan 31 03:08:28 dist-test-slave-5nnp krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828908, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:28.950616 27143 init.cc:377] Logged in from keytab as oryx/127.19.84.60@KRBTEST.COM (short username oryx)
I20260131 03:08:28.950969 27143 webserver.cc:492] Webserver started at http://127.19.84.60:41661/ using document root <none> and password file <none>
I20260131 03:08:28.951102 27143 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:28.951148 27143 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:28.951894 27143 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:28.952522 27325 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:28.952911 27143 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:28.952961 27143 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/wal
uuid: "005eb390434c4f61a808cc442be89188"
format_stamp: "Formatted at 2026-01-31 03:08:28 on dist-test-slave-5nnp"
server_key: "63a3a07a0faeb80786f398020cbb572e"
server_key_iv: "22ddcd0ee55ca85531a2391dcd8dca9a"
server_key_version: "encryptionkey@0"
I20260131 03:08:28.953009 27143 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:28.953117 27304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.953011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:36003 (local address 127.0.0.1:42294)
0131 03:08:28.953054 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:28.953079 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:36003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260131 03:08:28.953301 26540 master.cc:679] Network error: unable to get registration information for peer 005eb390434c4f61a808cc442be89188: Client connection negotiation failed: client connection to 127.19.84.60:36003: connect: Connection refused (error 111)
I20260131 03:08:28.976043 27143 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:28.977638 27143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:28.980818 27143 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.60:36003
I20260131 03:08:28.980847 27377 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.60:36003 every 8 connection(s)
I20260131 03:08:28.981144 27143 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/master-2/data/info.pb
I20260131 03:08:28.982009 27378 sys_catalog.cc:263] Verifying existing consensus state
I20260131 03:08:28.982545 27378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188: Bootstrap starting.
I20260131 03:08:28.985814 27378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260131 03:08:28.986232 27378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188: Bootstrap complete.
I20260131 03:08:28.986522 27378 raft_consensus.cc:359] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: NON_VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: true } }
I20260131 03:08:28.986617 27378 raft_consensus.cc:740] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 005eb390434c4f61a808cc442be89188, State: Initialized, Role: LEARNER
I20260131 03:08:28.986667 27378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: NON_VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: true } }
I20260131 03:08:28.986860 27381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: NON_VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: true } } }
I20260131 03:08:28.986989 27381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [sys.catalog]: This master's current role is: LEARNER
I20260131 03:08:28.986866 27378 sys_catalog.cc:565] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:08:28.988912 27393 catalog_manager.cc:1269] Loaded cluster ID: 00b8836497954b1f8f0752703e43dd26
I20260131 03:08:28.989071 27393 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188: loading cluster ID for follower catalog manager: success
I20260131 03:08:28.990278 27393 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188: acquiring CA information for follower catalog manager: success
I20260131 03:08:28.990609 27393 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260131 03:08:28.994307 27382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.987347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42310 (local address 127.19.84.60:36003)
0131 03:08:28.987596 (+ 249us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:28.987598 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:28.987612 (+ 14us) server_negotiation.cc:408] Connection header received
0131 03:08:28.987676 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:28.987678 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:28.987694 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:28.987852 (+ 158us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:28.988392 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.989772 (+ 1380us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.990585 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.990770 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.991752 (+ 982us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:28.991756 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:28.991757 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:28.991764 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:28.993288 (+ 1524us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.993676 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.993677 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.993677 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.993717 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:28.993878 (+ 161us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:28.993879 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:28.993879 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:28.993956 (+ 77us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:28.994003 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:28.994177 (+ 174us) server_negotiation.cc:300] Negotiation successful
0131 03:08:28.994247 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":201,"threads_started":1}
I20260131 03:08:28.994354 27304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:28.987308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:36003 (local address 127.0.0.1:42310)
0131 03:08:28.987353 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:28.987366 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:28.987440 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:28.987860 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:28.987861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:28.988132 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:28.988287 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.988290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.989873 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:28.989874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:28.990428 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:28.990430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:28.990506 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:28.991228 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:28.991233 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:28.991634 (+ 401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:28.993378 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.993379 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.993380 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.993593 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.993780 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:28.993781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:28.993782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:28.993819 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:28.994015 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:28.994016 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:28.994066 (+ 50us) client_negotiation.cc:769] Sending connection context
0131 03:08:28.994155 (+ 89us) client_negotiation.cc:240] Negotiation successful
0131 03:08:28.994262 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260131 03:08:29.072499 26741 heartbeater.cc:499] Master 127.19.84.62:33805 was elected leader, sending a full tablet report...
I20260131 03:08:29.251349 26877 heartbeater.cc:499] Master 127.19.84.62:33805 was elected leader, sending a full tablet report...
I20260131 03:08:29.291162 27352 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260131 03:08:29.291421 27381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c41b3e9bc84467e9d9758cbd2691e72. Latest consensus state: current_term: 1 leader_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: NON_VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: true } } }
I20260131 03:08:29.291538 27381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [sys.catalog]: This master's current role is: LEARNER
I20260131 03:08:29.422015 27013 heartbeater.cc:499] Master 127.19.84.62:33805 was elected leader, sending a full tablet report...
I20260131 03:08:29.738701 27117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: NON_VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: true } } }
I20260131 03:08:29.738867 27117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:29.739225 27117 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72: attempting to promote NON_VOTER 005eb390434c4f61a808cc442be89188 to VOTER
I20260131 03:08:29.739439 27117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: false } }
I20260131 03:08:29.739934 27352 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 1 LEARNER]: Refusing update from remote peer 6c41b3e9bc84467e9d9758cbd2691e72: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260131 03:08:29.739928 27170 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 1 FOLLOWER]: Refusing update from remote peer 6c41b3e9bc84467e9d9758cbd2691e72: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260131 03:08:29.740171 27395 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260131 03:08:29.740576 27117 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260131 03:08:29.741521 27395 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 005eb390434c4f61a808cc442be89188 (127.19.84.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: false } } }
I20260131 03:08:29.741674 27170 raft_consensus.cc:2955] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 005eb390434c4f61a808cc442be89188 (127.19.84.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: false } } }
I20260131 03:08:29.741938 27352 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 005eb390434c4f61a808cc442be89188 (127.19.84.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: false } } }
I20260131 03:08:29.742648 27395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: false } } }
I20260131 03:08:29.742738 27395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:29.742718 27117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: false } } }
I20260131 03:08:29.742774 27117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:29.742648 27394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: false } } }
I20260131 03:08:29.742853 27394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:29.742897 27398 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: false } } }
I20260131 03:08:29.742997 27398 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:08:29.743057 27381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: false } } }
I20260131 03:08:29.743144 27381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:08:29.970892 27401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:29.965219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34694 (local address 127.19.84.61:38999)
0131 03:08:29.965360 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:29.965362 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:29.965380 (+ 18us) server_negotiation.cc:408] Connection header received
0131 03:08:29.965402 (+ 22us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:29.965403 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:29.965413 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:29.965472 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:29.966177 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.966712 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:29.967266 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.967414 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:29.968543 (+ 1129us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:29.968548 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:29.968549 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:29.968555 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:29.969950 (+ 1395us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:29.970261 (+ 311us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:29.970262 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:29.970262 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:29.970282 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:29.970411 (+ 129us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:29.970411 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:29.970411 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:29.970500 (+ 89us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:29.970574 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:29.970673 (+ 99us) server_negotiation.cc:300] Negotiation successful
0131 03:08:29.970745 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260131 03:08:29.973379 27402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:29.971450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.0.0.1:58444)
0131 03:08:29.971570 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:29.971580 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:29.971625 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:29.971787 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:29.971788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:29.971790 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:29.971947 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:29.971948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.972618 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:29.972619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:29.973208 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:29.973210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.973270 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:29.973274 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:08:29.973297 (+ 23us) client_negotiation.cc:240] Negotiation successful
0131 03:08:29.973322 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260131 03:08:29.973820 27403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:29.971496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58444 (local address 127.19.84.62:33805)
0131 03:08:29.971667 (+ 171us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:29.971669 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:29.971683 (+ 14us) server_negotiation.cc:408] Connection header received
0131 03:08:29.971707 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:29.971707 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:29.971717 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:29.971773 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:29.972015 (+ 242us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.972520 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:29.973299 (+ 779us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.973698 (+ 399us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:29.973710 (+ 12us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:29.973739 (+ 29us) server_negotiation.cc:300] Negotiation successful
0131 03:08:29.973765 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":118,"threads_started":1}
I20260131 03:08:29.976199 27402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:29.974498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:38999 (local address 127.0.0.1:34700)
0131 03:08:29.974565 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:29.974576 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:29.974621 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:29.974732 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:29.974733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:29.974734 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:29.974858 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:29.974860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.975506 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:29.975507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:29.976043 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:29.976044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.976103 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:29.976106 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:08:29.976129 (+ 23us) client_negotiation.cc:240] Negotiation successful
0131 03:08:29.976154 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260131 03:08:29.976609 27401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:29.974566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34700 (local address 127.19.84.61:38999)
0131 03:08:29.974614 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:29.974616 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:29.974623 (+ 7us) server_negotiation.cc:408] Connection header received
0131 03:08:29.974661 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:29.974662 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:29.974670 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:29.974715 (+ 45us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:29.974949 (+ 234us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.975411 (+ 462us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:29.976113 (+ 702us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.976519 (+ 406us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:29.976525 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:29.976542 (+ 17us) server_negotiation.cc:300] Negotiation successful
0131 03:08:29.976569 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260131 03:08:29.979046 27402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:29.977227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:36003 (local address 127.0.0.1:42312)
0131 03:08:29.977275 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:29.977288 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:29.977330 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:29.977593 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:29.977593 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:29.977595 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:29.977702 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:29.977702 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.978300 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:29.978300 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:29.978859 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:29.978861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.978933 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:29.978936 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:08:29.978968 (+ 32us) client_negotiation.cc:240] Negotiation successful
0131 03:08:29.979004 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260131 03:08:29.979627 27404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:29.977332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42312 (local address 127.19.84.60:36003)
0131 03:08:29.977469 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:29.977471 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:29.977479 (+ 8us) server_negotiation.cc:408] Connection header received
0131 03:08:29.977500 (+ 21us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:29.977500 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:29.977524 (+ 24us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:29.977585 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:29.977762 (+ 177us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.978241 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:29.978962 (+ 721us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.979422 (+ 460us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:29.979523 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:29.979544 (+ 21us) server_negotiation.cc:300] Negotiation successful
0131 03:08:29.979577 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":87,"threads_started":1}
I20260131 03:08:29.987404 27404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:29.980610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42320 (local address 127.19.84.60:36003)
0131 03:08:29.980655 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:29.980657 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:29.980747 (+ 90us) server_negotiation.cc:408] Connection header received
0131 03:08:29.980793 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:29.980794 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:29.980804 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:29.980868 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:29.981668 (+ 800us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.982144 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:29.982707 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.982907 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:29.983931 (+ 1024us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:29.983939 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:29.983939 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:29.983946 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:29.985978 (+ 2032us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:29.986365 (+ 387us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:29.986367 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:29.986367 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:29.986393 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:29.986721 (+ 328us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:29.986722 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:29.986722 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:29.986821 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:29.986907 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:29.987159 (+ 252us) server_negotiation.cc:300] Negotiation successful
0131 03:08:29.987254 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260131 03:08:29.990267 27405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:29.987952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.0.0.1:58454)
0131 03:08:29.988178 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:29.988190 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:29.988289 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:29.988452 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:29.988452 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:29.988456 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:29.988640 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:29.988641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.989365 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:29.989366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:29.989998 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:29.989999 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.990092 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:29.990098 (+ 6us) client_negotiation.cc:769] Sending connection context
0131 03:08:29.990131 (+ 33us) client_negotiation.cc:240] Negotiation successful
0131 03:08:29.990173 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":102,"threads_started":1}
I20260131 03:08:29.990675 27403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:29.988074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58454 (local address 127.19.84.62:33805)
0131 03:08:29.988175 (+ 101us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:29.988178 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:29.988271 (+ 93us) server_negotiation.cc:408] Connection header received
0131 03:08:29.988369 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:29.988370 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:29.988380 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:29.988451 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:29.988730 (+ 279us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.989287 (+ 557us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:29.990091 (+ 804us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.990567 (+ 476us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:29.990577 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:29.990602 (+ 25us) server_negotiation.cc:300] Negotiation successful
0131 03:08:29.990628 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260131 03:08:29.993427 27405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:29.991426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:38999 (local address 127.0.0.1:34712)
0131 03:08:29.991529 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:29.991567 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:29.991650 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:29.991838 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:29.991839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:29.991841 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:29.991986 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:29.991987 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.992615 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:29.992616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:29.993193 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:29.993195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.993287 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:29.993291 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:08:29.993324 (+ 33us) client_negotiation.cc:240] Negotiation successful
0131 03:08:29.993361 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260131 03:08:29.993875 27401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:29.991486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34712 (local address 127.19.84.61:38999)
0131 03:08:29.991529 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:29.991532 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:29.991635 (+ 103us) server_negotiation.cc:408] Connection header received
0131 03:08:29.991729 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:29.991730 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:29.991738 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:29.991822 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:29.992061 (+ 239us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.992524 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:29.993287 (+ 763us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.993753 (+ 466us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:29.993761 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:29.993793 (+ 32us) server_negotiation.cc:300] Negotiation successful
0131 03:08:29.993828 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260131 03:08:29.996318 27405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:29.994522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:36003 (local address 127.0.0.1:42330)
0131 03:08:29.994586 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:29.994598 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:29.994652 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:29.994751 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:29.994752 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:29.994754 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:29.994883 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:29.994884 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.995538 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:29.995539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:29.996099 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:29.996100 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.996181 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:29.996185 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:08:29.996220 (+ 35us) client_negotiation.cc:240] Negotiation successful
0131 03:08:29.996253 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260131 03:08:29.996814 27404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:29.994582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42330 (local address 127.19.84.60:36003)
0131 03:08:29.994614 (+ 32us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:29.994616 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:29.994640 (+ 24us) server_negotiation.cc:408] Connection header received
0131 03:08:29.994690 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:29.994691 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:29.994698 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:29.994739 (+ 41us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:29.994940 (+ 201us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.995449 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:29.996191 (+ 742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:29.996725 (+ 534us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:29.996732 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:29.996752 (+ 20us) server_negotiation.cc:300] Negotiation successful
0131 03:08:29.996774 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260131 03:08:29.997663 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 26612
I20260131 03:08:30.002724 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:35541
--local_ip_for_outbound_sockets=127.19.84.1
--tserver_master_addrs=127.19.84.62:33805
--webserver_port=34329
--webserver_interface=127.19.84.1
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:37697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.84.62:33805,127.19.84.61:38999
--tserver_master_addrs=127.19.84.62:33805,127.19.84.61:38999,127.19.84.60:36003 with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:30.086151 27406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:30.086355 27406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:30.086381 27406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:30.087973 27406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:30.088021 27406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:30.088050 27406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:08:30.089813 27406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:37697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:35541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=34329
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:33805,127.19.84.61:38999,127.19.84.60:36003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:30.090070 27406 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:30.090291 27406 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260131 03:08:30.093209 27406 server_base.cc:1060] running on GCE node
W20260131 03:08:30.093158 27411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:30.093158 27412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:30.093163 27414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:30.093595 27406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:30.093860 27406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:30.095045 27406 hybrid_clock.cc:648] HybridClock initialized: now 1769828910094961 us; error 101 us; skew 500 ppm
Jan 31 03:08:30 dist-test-slave-5nnp krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828910, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:30.097810 27406 init.cc:377] Logged in from keytab as oryx/127.19.84.1@KRBTEST.COM (short username oryx)
I20260131 03:08:30.098397 27406 webserver.cc:492] Webserver started at http://127.19.84.1:34329/ using document root <none> and password file <none>
I20260131 03:08:30.098626 27406 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:30.098726 27406 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:30.100242 27406 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:08:30.101024 27421 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:30.101380 27406 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:08:30.101495 27406 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "0df52ee4e3d4496baaf359d610b7d34c"
format_stamp: "Formatted at 2026-01-31 03:08:28 on dist-test-slave-5nnp"
server_key: "55abfd4d9937bcda1a06e3abd5878b99"
server_key_iv: "8d918f58c3a661b761b5cf83b788c569"
server_key_version: "encryptionkey@0"
I20260131 03:08:30.101815 27406 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:30.115208 27406 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:30.117328 27406 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:30.117442 27406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:30.117624 27406 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:30.117964 27406 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:30.118000 27406 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:30.118022 27406 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:30.118036 27406 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:30.124167 27406 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:35541
I20260131 03:08:30.124207 27535 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:35541 every 8 connection(s)
I20260131 03:08:30.124536 27406 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
I20260131 03:08:30.127540 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 27406
I20260131 03:08:30.127697 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 26748
Jan 31 03:08:30 dist-test-slave-5nnp krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828910, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM
Jan 31 03:08:30 dist-test-slave-5nnp krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828910, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.61@KRBTEST.COM
Jan 31 03:08:30 dist-test-slave-5nnp krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828910, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.60@KRBTEST.COM
I20260131 03:08:30.133674 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:39245
--local_ip_for_outbound_sockets=127.19.84.2
--tserver_master_addrs=127.19.84.62:33805
--webserver_port=37775
--webserver_interface=127.19.84.2
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:37697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.84.62:33805,127.19.84.61:38999
--tserver_master_addrs=127.19.84.62:33805,127.19.84.61:38999,127.19.84.60:36003 with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260131 03:08:30.137080 27403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.125274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:45043 (local address 127.19.84.62:33805)
0131 03:08:30.125324 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:30.125326 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:30.125874 (+ 548us) server_negotiation.cc:408] Connection header received
0131 03:08:30.126139 (+ 265us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:30.126141 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:30.126153 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:30.126185 (+ 32us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:30.127486 (+ 1301us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.128283 (+ 797us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.129567 (+ 1284us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.129738 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.132988 (+ 3250us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:30.132993 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:30.132993 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:30.133001 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:30.135489 (+ 2488us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.135999 (+ 510us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.136000 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.136001 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.136031 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.136286 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.136288 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.136289 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.136440 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:30.136487 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:30.136884 (+ 397us) server_negotiation.cc:300] Negotiation successful
0131 03:08:30.136981 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260131 03:08:30.137080 27401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.125429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:33137 (local address 127.19.84.61:38999)
0131 03:08:30.125505 (+ 76us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:30.125507 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:30.125583 (+ 76us) server_negotiation.cc:408] Connection header received
0131 03:08:30.126139 (+ 556us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:30.126141 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:30.126153 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:30.126185 (+ 32us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:30.127269 (+ 1084us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.128028 (+ 759us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.129882 (+ 1854us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.130065 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.132990 (+ 2925us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:30.132993 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:30.132993 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:30.133002 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:30.135948 (+ 2946us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.136316 (+ 368us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.136317 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.136317 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.136340 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.136538 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.136539 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.136539 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.136652 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:30.136692 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:30.136879 (+ 187us) server_negotiation.cc:300] Negotiation successful
0131 03:08:30.136980 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260131 03:08:30.137531 27540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.125223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.1:45043)
0131 03:08:30.125798 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:30.125806 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:30.126041 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:30.126267 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:30.126267 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:30.126755 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:30.127359 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.127361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.129046 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.129047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:30.129458 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.129459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.129622 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.130342 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:30.130361 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:30.132877 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:30.135579 (+ 2702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:30.135587 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:30.135590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:30.135907 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:30.136106 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:30.136107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:30.136107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:30.136195 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:30.136536 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:30.136541 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:30.136708 (+ 167us) client_negotiation.cc:769] Sending connection context
0131 03:08:30.137174 (+ 466us) client_negotiation.cc:240] Negotiation successful
0131 03:08:30.137313 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":187,"thread_start_us":50,"threads_started":1}
I20260131 03:08:30.137542 27542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.125222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:38999 (local address 127.19.84.1:33137)
0131 03:08:30.125525 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:30.125535 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:30.126041 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:30.126266 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:30.126267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:30.126761 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:30.127126 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.127129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.129117 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.129119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:30.129788 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.129791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.129869 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.130329 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:30.130358 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:30.132877 (+ 2519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:30.136032 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:30.136033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:30.136033 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:30.136231 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:30.136408 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:30.136408 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:30.136409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:30.136456 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:30.136737 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:30.136738 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:30.136770 (+ 32us) client_negotiation.cc:769] Sending connection context
0131 03:08:30.137197 (+ 427us) client_negotiation.cc:240] Negotiation successful
0131 03:08:30.137367 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":92,"thread_start_us":63,"threads_started":1}
I20260131 03:08:30.137959 27538 heartbeater.cc:344] Connected to a master server at 127.19.84.61:38999
I20260131 03:08:30.137970 27536 heartbeater.cc:344] Connected to a master server at 127.19.84.62:33805
I20260131 03:08:30.138067 27538 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:30.138067 27536 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:30.138317 27536 heartbeater.cc:507] Master 127.19.84.62:33805 requested a full tablet report, sending...
I20260131 03:08:30.138342 27541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.125222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:36003 (local address 127.19.84.1:46573)
0131 03:08:30.125422 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:30.125488 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:30.126023 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:30.126263 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:30.126267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:30.126755 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:30.127059 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.127062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.130044 (+ 2982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.130045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:30.131442 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.131445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.131514 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.131957 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:30.131962 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:30.133889 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:30.136252 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:30.136253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:30.136254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:30.136469 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:30.136645 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:30.136645 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:30.136646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:30.136674 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:30.136948 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:30.136949 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:30.136992 (+ 43us) client_negotiation.cc:769] Sending connection context
0131 03:08:30.137070 (+ 78us) client_negotiation.cc:240] Negotiation successful
0131 03:08:30.137227 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":76,"threads_started":1}
I20260131 03:08:30.138317 27538 heartbeater.cc:507] Master 127.19.84.61:38999 requested a full tablet report, sending...
I20260131 03:08:30.138622 27404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.125324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:46573 (local address 127.19.84.60:36003)
0131 03:08:30.125362 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:30.125364 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:30.125642 (+ 278us) server_negotiation.cc:408] Connection header received
0131 03:08:30.126125 (+ 483us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:30.126126 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:30.126138 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:30.126198 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:30.127162 (+ 964us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.127951 (+ 789us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.132432 (+ 4481us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.132636 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.134001 (+ 1365us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:30.134006 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:30.134006 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:30.134013 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:30.136177 (+ 2164us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.136553 (+ 376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.136554 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.136555 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.136580 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.136734 (+ 154us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.136735 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.136735 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.136864 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:30.136922 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:30.138402 (+ 1480us) server_negotiation.cc:300] Negotiation successful
0131 03:08:30.138507 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260131 03:08:30.138710 26540 ts_manager.cc:194] Re-registered known tserver with Master: 0df52ee4e3d4496baaf359d610b7d34c (127.19.84.1:35541)
I20260131 03:08:30.138810 27160 ts_manager.cc:194] Registered new tserver with Master: 0df52ee4e3d4496baaf359d610b7d34c (127.19.84.1:35541)
I20260131 03:08:30.138947 27537 heartbeater.cc:344] Connected to a master server at 127.19.84.60:36003
I20260131 03:08:30.139008 27537 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:30.139096 27537 heartbeater.cc:507] Master 127.19.84.60:36003 requested a full tablet report, sending...
I20260131 03:08:30.139256 26540 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.84.1@KRBTEST.COM'} at 127.19.84.1:45043
I20260131 03:08:30.139516 27342 ts_manager.cc:194] Registered new tserver with Master: 0df52ee4e3d4496baaf359d610b7d34c (127.19.84.1:35541)
W20260131 03:08:30.219769 27543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:30.220033 27543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:30.220075 27543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:30.221616 27543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:30.221693 27543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:30.221740 27543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:08:30.223536 27543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:37697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:39245
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=37775
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:33805,127.19.84.61:38999,127.19.84.60:36003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:30.223803 27543 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:30.224036 27543 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:30.227061 27548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:30.227209 27543 server_base.cc:1060] running on GCE node
W20260131 03:08:30.227056 27549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:30.227056 27551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:30.227488 27543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:30.227766 27543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:30.228943 27543 hybrid_clock.cc:648] HybridClock initialized: now 1769828910228885 us; error 76 us; skew 500 ppm
Jan 31 03:08:30 dist-test-slave-5nnp krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828910, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:30.231436 27543 init.cc:377] Logged in from keytab as oryx/127.19.84.2@KRBTEST.COM (short username oryx)
I20260131 03:08:30.231977 27543 webserver.cc:492] Webserver started at http://127.19.84.2:37775/ using document root <none> and password file <none>
I20260131 03:08:30.232187 27543 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:30.232259 27543 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:30.233646 27543 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:30.234256 27558 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:30.234475 27543 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:30.234587 27543 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6"
format_stamp: "Formatted at 2026-01-31 03:08:28 on dist-test-slave-5nnp"
server_key: "a344c4481a47f0f454483959b448195b"
server_key_iv: "96800618cf59505e9e470cc86fee9090"
server_key_version: "encryptionkey@0"
I20260131 03:08:30.234895 27543 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:30.253019 27543 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:30.255134 27543 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:30.255306 27543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:30.255585 27543 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:30.255955 27543 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:30.256013 27543 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:30.256058 27543 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:30.256083 27543 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:30.262223 27543 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:39245
I20260131 03:08:30.262306 27671 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:39245 every 8 connection(s)
I20260131 03:08:30.262624 27543 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
Jan 31 03:08:30 dist-test-slave-5nnp krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828910, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.60@KRBTEST.COM
Jan 31 03:08:30 dist-test-slave-5nnp krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828910, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM
Jan 31 03:08:30 dist-test-slave-5nnp krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828910, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.61@KRBTEST.COM
I20260131 03:08:30.270789 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 27543
I20260131 03:08:30.270936 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 26884
I20260131 03:08:30.272706 27404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.263491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:55799 (local address 127.19.84.60:36003)
0131 03:08:30.263578 (+ 87us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:30.263580 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:30.263747 (+ 167us) server_negotiation.cc:408] Connection header received
0131 03:08:30.264385 (+ 638us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:30.264387 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:30.264399 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:30.264432 (+ 33us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:30.265607 (+ 1175us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.266109 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.266716 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.266884 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.269312 (+ 2428us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:30.269316 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:30.269317 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:30.269323 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:30.271100 (+ 1777us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.271591 (+ 491us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.271593 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.271593 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.271620 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.271857 (+ 237us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.271858 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.271859 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.271982 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:30.272039 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:30.272445 (+ 406us) server_negotiation.cc:300] Negotiation successful
0131 03:08:30.272549 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260131 03:08:30.272771 27676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.263312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:36003 (local address 127.19.84.2:55799)
0131 03:08:30.263637 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:30.263672 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:30.264273 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:30.264564 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:30.264565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:30.265046 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:30.265500 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.265504 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.266202 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.266203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:30.266617 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.266618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.266723 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.267357 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:30.267407 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:30.269227 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:30.271181 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:30.271195 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:30.271198 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:30.271507 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:30.271694 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:30.271695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:30.271696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:30.271783 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:30.272077 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:30.272083 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:30.272251 (+ 168us) client_negotiation.cc:769] Sending connection context
0131 03:08:30.272380 (+ 129us) client_negotiation.cc:240] Negotiation successful
0131 03:08:30.272546 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":95,"threads_started":1}
I20260131 03:08:30.274124 27401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.263813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:49739 (local address 127.19.84.61:38999)
0131 03:08:30.263851 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:30.263853 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:30.263860 (+ 7us) server_negotiation.cc:408] Connection header received
0131 03:08:30.264368 (+ 508us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:30.264369 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:30.264381 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:30.264440 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:30.266809 (+ 2369us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.267581 (+ 772us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.268285 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.268522 (+ 237us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.270413 (+ 1891us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:30.270417 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:30.270417 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:30.270425 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:30.272973 (+ 2548us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.273284 (+ 311us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.273285 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.273286 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.273309 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.273605 (+ 296us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.273607 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.273607 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.273714 (+ 107us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:30.273795 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:30.273952 (+ 157us) server_negotiation.cc:300] Negotiation successful
0131 03:08:30.274035 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260131 03:08:30.274139 27678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.263312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:38999 (local address 127.19.84.2:49739)
0131 03:08:30.263719 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:30.263730 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:30.264266 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:30.264522 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:30.264527 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:30.265052 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:30.265508 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.265510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.267666 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.267667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:30.268201 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.268204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.268260 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.268961 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:30.268965 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:30.270307 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:30.273059 (+ 2752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:30.273060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:30.273060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:30.273202 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:30.273365 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:30.273366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:30.273366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:30.273392 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:30.273787 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:30.273788 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:30.273833 (+ 45us) client_negotiation.cc:769] Sending connection context
0131 03:08:30.273915 (+ 82us) client_negotiation.cc:240] Negotiation successful
0131 03:08:30.274029 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":113,"thread_start_us":118,"threads_started":1}
I20260131 03:08:30.274492 27674 heartbeater.cc:344] Connected to a master server at 127.19.84.61:38999
I20260131 03:08:30.274616 27674 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:30.274837 27674 heartbeater.cc:507] Master 127.19.84.61:38999 requested a full tablet report, sending...
I20260131 03:08:30.275291 27160 ts_manager.cc:194] Registered new tserver with Master: 33d5b3c3b9d74484b9a5f92b2dc57fa6 (127.19.84.2:39245)
I20260131 03:08:30.276590 27673 heartbeater.cc:344] Connected to a master server at 127.19.84.60:36003
I20260131 03:08:30.276657 27673 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:30.276791 27673 heartbeater.cc:507] Master 127.19.84.60:36003 requested a full tablet report, sending...
I20260131 03:08:30.277124 27342 ts_manager.cc:194] Registered new tserver with Master: 33d5b3c3b9d74484b9a5f92b2dc57fa6 (127.19.84.2:39245)
I20260131 03:08:30.277640 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:44133
--local_ip_for_outbound_sockets=127.19.84.3
--tserver_master_addrs=127.19.84.62:33805
--webserver_port=43883
--webserver_interface=127.19.84.3
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:37697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.84.62:33805,127.19.84.61:38999
--tserver_master_addrs=127.19.84.62:33805,127.19.84.61:38999,127.19.84.60:36003 with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260131 03:08:30.278051 27403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.263661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:39677 (local address 127.19.84.62:33805)
0131 03:08:30.263720 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:30.263722 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:30.263739 (+ 17us) server_negotiation.cc:408] Connection header received
0131 03:08:30.264385 (+ 646us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:30.264387 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:30.264400 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:30.264432 (+ 32us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:30.266197 (+ 1765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.266694 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.267414 (+ 720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.267823 (+ 409us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.270398 (+ 2575us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:30.270402 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:30.270403 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:30.270410 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:30.276976 (+ 6566us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.277340 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.277341 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.277341 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.277367 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.277536 (+ 169us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.277537 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.277537 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.277636 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:30.277706 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:30.277900 (+ 194us) server_negotiation.cc:300] Negotiation successful
0131 03:08:30.277990 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260131 03:08:30.278311 27677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.263312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.2:39677)
0131 03:08:30.263637 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:30.263672 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:30.264278 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:30.264524 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:30.264527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:30.265046 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:30.265500 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.265504 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.266796 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.266797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:30.267276 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.267278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.267367 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.268440 (+ 1073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:30.268444 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:30.270311 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:30.277059 (+ 6748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:30.277060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:30.277061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:30.277271 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:30.277449 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:30.277450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:30.277451 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:30.277479 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:30.277708 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:30.277709 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:30.277772 (+ 63us) client_negotiation.cc:769] Sending connection context
0131 03:08:30.278151 (+ 379us) client_negotiation.cc:240] Negotiation successful
0131 03:08:30.278245 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":3,"thread_start_us":114,"threads_started":1}
I20260131 03:08:30.278692 27672 heartbeater.cc:344] Connected to a master server at 127.19.84.62:33805
I20260131 03:08:30.278759 27672 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:30.278870 27672 heartbeater.cc:507] Master 127.19.84.62:33805 requested a full tablet report, sending...
I20260131 03:08:30.279125 26540 ts_manager.cc:194] Re-registered known tserver with Master: 33d5b3c3b9d74484b9a5f92b2dc57fa6 (127.19.84.2:39245)
I20260131 03:08:30.279706 26540 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.84.2@KRBTEST.COM'} at 127.19.84.2:39677
W20260131 03:08:30.361644 27679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:30.361852 27679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:30.361878 27679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:30.363483 27679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:30.363543 27679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:30.363576 27679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:08:30.365501 27679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:37697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:44133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=43883
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:33805,127.19.84.61:38999,127.19.84.60:36003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:30.365751 27679 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:30.365944 27679 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:30.368897 27685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:30.368919 27684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:30.368927 27687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:30.369092 27679 server_base.cc:1060] running on GCE node
I20260131 03:08:30.369352 27679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:30.369575 27679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:30.370733 27679 hybrid_clock.cc:648] HybridClock initialized: now 1769828910370699 us; error 68 us; skew 500 ppm
Jan 31 03:08:30 dist-test-slave-5nnp krb5kdc[26500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828910, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:30.373212 27679 init.cc:377] Logged in from keytab as oryx/127.19.84.3@KRBTEST.COM (short username oryx)
I20260131 03:08:30.373766 27679 webserver.cc:492] Webserver started at http://127.19.84.3:43883/ using document root <none> and password file <none>
I20260131 03:08:30.373970 27679 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:30.374017 27679 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:30.375312 27679 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260131 03:08:30.375994 27694 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:30.376174 27679 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260131 03:08:30.376259 27679 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "e174c1a5c18148c5ac26d4824b84fe49"
format_stamp: "Formatted at 2026-01-31 03:08:28 on dist-test-slave-5nnp"
server_key: "0df258b4289d04243c2088951e8e82d2"
server_key_iv: "d56e26f0e6ddc237e482fd61454e74f4"
server_key_version: "encryptionkey@0"
I20260131 03:08:30.376606 27679 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:30.387809 27679 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:30.389797 27679 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:30.389957 27679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:30.390178 27679 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:30.390487 27679 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:30.390548 27679 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:30.390594 27679 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:30.390625 27679 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:30.397079 27679 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:44133
I20260131 03:08:30.397148 27807 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:44133 every 8 connection(s)
I20260131 03:08:30.397486 27679 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
I20260131 03:08:30.402734 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 27679
Jan 31 03:08:30 dist-test-slave-5nnp krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828910, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.61@KRBTEST.COM
Jan 31 03:08:30 dist-test-slave-5nnp krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828910, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.60@KRBTEST.COM
Jan 31 03:08:30 dist-test-slave-5nnp krb5kdc[26500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828910, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM
I20260131 03:08:30.409099 27813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.398208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.3:35089)
0131 03:08:30.399071 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:30.399078 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:30.399151 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:30.399307 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:30.399308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:30.399811 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:30.400206 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.400209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.401169 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.401169 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:30.401727 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.401729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.401812 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.402241 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:30.402246 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:30.405527 (+ 3281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:30.407707 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:30.407708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:30.407708 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:30.407911 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:30.408086 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:30.408087 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:30.408088 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:30.408182 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:30.408436 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:30.408441 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:30.408602 (+ 161us) client_negotiation.cc:769] Sending connection context
0131 03:08:30.408717 (+ 115us) client_negotiation.cc:240] Negotiation successful
0131 03:08:30.408867 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"thread_start_us":75,"threads_started":1}
I20260131 03:08:30.409313 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.398207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:36003 (local address 127.19.84.3:36319)
0131 03:08:30.398355 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:30.398390 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:30.398939 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:30.399195 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:30.399198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:30.399811 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:30.400443 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.400445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.401310 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.401311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:30.401839 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.401841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.402156 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.402547 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:30.402551 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:30.405224 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:30.408282 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:30.408283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:30.408284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:30.408471 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:30.408669 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:30.408670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:30.408671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:30.408705 (+ 34us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:30.408985 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:30.408986 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:30.409027 (+ 41us) client_negotiation.cc:769] Sending connection context
0131 03:08:30.409094 (+ 67us) client_negotiation.cc:240] Negotiation successful
0131 03:08:30.409199 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":94,"threads_started":1}
I20260131 03:08:30.409425 27403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.398304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:35089 (local address 127.19.84.62:33805)
0131 03:08:30.398415 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:30.398417 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:30.399154 (+ 737us) server_negotiation.cc:408] Connection header received
0131 03:08:30.399213 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:30.399214 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:30.399227 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:30.399288 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:30.400332 (+ 1044us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.401076 (+ 744us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.401827 (+ 751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.402013 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.405620 (+ 3607us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:30.405624 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:30.405625 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:30.405633 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:30.407329 (+ 1696us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.407997 (+ 668us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.407998 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.407999 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.408023 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.408252 (+ 229us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.408253 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.408253 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.408358 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:30.408414 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:30.409246 (+ 832us) server_negotiation.cc:300] Negotiation successful
0131 03:08:30.409342 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260131 03:08:30.409703 27404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.398368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:36319 (local address 127.19.84.60:36003)
0131 03:08:30.398448 (+ 80us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:30.398450 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:30.398514 (+ 64us) server_negotiation.cc:408] Connection header received
0131 03:08:30.399027 (+ 513us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:30.399028 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:30.399038 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:30.399121 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:30.400547 (+ 1426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.401230 (+ 683us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.401935 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.402083 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.405314 (+ 3231us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:30.405318 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:30.405318 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:30.405327 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:30.407873 (+ 2546us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.408569 (+ 696us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.408570 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.408571 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.408597 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.408772 (+ 175us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.408772 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.408773 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.408898 (+ 125us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:30.408959 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:30.409533 (+ 574us) server_negotiation.cc:300] Negotiation successful
0131 03:08:30.409624 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260131 03:08:30.409765 27808 heartbeater.cc:344] Connected to a master server at 127.19.84.62:33805
I20260131 03:08:30.409860 27808 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:30.409981 27809 heartbeater.cc:344] Connected to a master server at 127.19.84.60:36003
I20260131 03:08:30.410023 27809 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:30.410092 27808 heartbeater.cc:507] Master 127.19.84.62:33805 requested a full tablet report, sending...
I20260131 03:08:30.410092 27809 heartbeater.cc:507] Master 127.19.84.60:36003 requested a full tablet report, sending...
I20260131 03:08:30.410494 26540 ts_manager.cc:194] Re-registered known tserver with Master: e174c1a5c18148c5ac26d4824b84fe49 (127.19.84.3:44133)
I20260131 03:08:30.410597 27342 ts_manager.cc:194] Registered new tserver with Master: e174c1a5c18148c5ac26d4824b84fe49 (127.19.84.3:44133)
I20260131 03:08:30.410997 26540 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:35089
I20260131 03:08:30.411499 27814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.398513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:38999 (local address 127.19.84.3:49593)
0131 03:08:30.399234 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:30.399242 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:30.399320 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:30.399500 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:30.399500 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:30.399819 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:30.400178 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.400182 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.400925 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.400926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:30.401329 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.401330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.401488 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.402122 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:30.402148 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:30.404178 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:30.406603 (+ 2425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:30.406609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:30.406612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:30.406922 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:30.410094 (+ 3172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:30.410095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:30.410096 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:30.410130 (+ 34us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:30.410691 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:30.410692 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:30.410738 (+ 46us) client_negotiation.cc:769] Sending connection context
0131 03:08:30.410927 (+ 189us) client_negotiation.cc:240] Negotiation successful
0131 03:08:30.411331 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":61,"threads_started":1}
I20260131 03:08:30.411935 27401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.398585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:49593 (local address 127.19.84.61:38999)
0131 03:08:30.398630 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:30.398632 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:30.399292 (+ 660us) server_negotiation.cc:408] Connection header received
0131 03:08:30.399404 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:30.399405 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:30.399417 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:30.399494 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:30.400337 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.400829 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.401424 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.401612 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.404317 (+ 2705us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:30.404322 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:30.404322 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:30.404330 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:30.406373 (+ 2043us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.409993 (+ 3620us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.409993 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.409994 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.410024 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.410197 (+ 173us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.410198 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.410199 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.410584 (+ 385us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:30.410658 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:30.411744 (+ 1086us) server_negotiation.cc:300] Negotiation successful
0131 03:08:30.411832 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260131 03:08:30.412230 27810 heartbeater.cc:344] Connected to a master server at 127.19.84.61:38999
I20260131 03:08:30.412285 27810 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:30.412343 27810 heartbeater.cc:507] Master 127.19.84.61:38999 requested a full tablet report, sending...
I20260131 03:08:30.412673 27160 ts_manager.cc:194] Registered new tserver with Master: e174c1a5c18148c5ac26d4824b84fe49 (127.19.84.3:44133)
I20260131 03:08:30.413923 27820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.404060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.19.84.61:38999)
0131 03:08:30.404420 (+ 360us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:30.404422 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:30.404618 (+ 196us) server_negotiation.cc:408] Connection header received
0131 03:08:30.404641 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:30.404642 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:30.404655 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:30.404712 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:30.405758 (+ 1046us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.406594 (+ 836us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.407508 (+ 914us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.407704 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.409989 (+ 2285us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:30.409993 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:30.409993 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:30.410001 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:30.412633 (+ 2632us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.413053 (+ 420us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.413054 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.413054 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.413078 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.413346 (+ 268us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.413346 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.413347 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.413478 (+ 131us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:30.413590 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:30.413745 (+ 155us) server_negotiation.cc:300] Negotiation successful
0131 03:08:30.413831 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":64,"threads_started":1}
I20260131 03:08:30.415163 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.404172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42334 (local address 127.19.84.60:36003)
0131 03:08:30.404334 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:30.404336 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:30.407088 (+ 2752us) server_negotiation.cc:408] Connection header received
0131 03:08:30.407115 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:30.407116 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:30.407130 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:30.407185 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:30.408132 (+ 947us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.408867 (+ 735us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.410465 (+ 1598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.410644 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.412133 (+ 1489us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:30.412136 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:30.412136 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:30.412143 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:30.414234 (+ 2091us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.414518 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.414519 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.414519 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.414542 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.414650 (+ 108us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.414651 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.414651 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.414755 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:30.414859 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:30.414962 (+ 103us) server_negotiation.cc:300] Negotiation successful
0131 03:08:30.415058 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":52,"threads_started":1}
I20260131 03:08:30.418854 27403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.407615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.19.84.62:33805)
0131 03:08:30.409495 (+ 1880us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:30.409497 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:30.411062 (+ 1565us) server_negotiation.cc:408] Connection header received
0131 03:08:30.411185 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:30.411185 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:30.411195 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:30.411223 (+ 28us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:30.413411 (+ 2188us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.414150 (+ 739us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.414909 (+ 759us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.415095 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.416574 (+ 1479us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:30.416578 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:30.416578 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:30.416585 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:30.417923 (+ 1338us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.418211 (+ 288us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.418212 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.418212 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.418231 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.418393 (+ 162us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.418393 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.418394 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.418469 (+ 75us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:30.418534 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:30.418702 (+ 168us) server_negotiation.cc:300] Negotiation successful
0131 03:08:30.418771 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1864,"thread_start_us":52,"threads_started":1}
I20260131 03:08:30.420409 26540 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:08:30.421680 26540 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:08:30.428820 27833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.425754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:39245 (local address 127.0.0.1:37478)
0131 03:08:30.426137 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:30.426146 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:30.426200 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:30.426385 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:30.426386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:30.426388 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:30.426598 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.426600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.427680 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.427681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:30.428563 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.428565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.428642 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.428648 (+ 6us) client_negotiation.cc:769] Sending connection context
0131 03:08:30.428681 (+ 33us) client_negotiation.cc:240] Negotiation successful
0131 03:08:30.428723 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":71,"threads_started":1}
I20260131 03:08:30.429489 27835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.425828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49414 (local address 127.19.84.1:35541)
0131 03:08:30.426236 (+ 408us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:30.426243 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:30.426254 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:08:30.426295 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:30.426297 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:30.426315 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:30.426392 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:30.426727 (+ 335us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.427815 (+ 1088us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.428772 (+ 957us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.429272 (+ 500us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.429329 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:30.429364 (+ 35us) server_negotiation.cc:300] Negotiation successful
0131 03:08:30.429403 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":75,"threads_started":1}
I20260131 03:08:30.429505 27837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.425953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37478 (local address 127.19.84.2:39245)
0131 03:08:30.426111 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:30.426120 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:30.426192 (+ 72us) server_negotiation.cc:408] Connection header received
0131 03:08:30.426267 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:30.426269 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:30.426288 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:30.426364 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:30.426682 (+ 318us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.427597 (+ 915us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.428651 (+ 1054us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.429314 (+ 663us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.429349 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:30.429394 (+ 45us) server_negotiation.cc:300] Negotiation successful
0131 03:08:30.429435 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260131 03:08:30.428920 27834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.425711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:35541 (local address 127.0.0.1:49414)
0131 03:08:30.425982 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:30.425996 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:30.426057 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:30.426398 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:30.426399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:30.426402 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:30.426633 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.426636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.427895 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.427896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:30.428692 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.428693 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.428759 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.428762 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:08:30.428793 (+ 31us) client_negotiation.cc:240] Negotiation successful
0131 03:08:30.428823 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":91,"threads_started":1}
I20260131 03:08:30.429867 27836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.425909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:44133 (local address 127.0.0.1:56610)
0131 03:08:30.426588 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:30.426597 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:30.426649 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:30.426857 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:30.426858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:30.426861 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:30.427047 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.427048 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.429002 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.429003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:30.429663 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.429667 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.429740 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.429744 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:08:30.429769 (+ 25us) client_negotiation.cc:240] Negotiation successful
0131 03:08:30.429802 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":75,"threads_started":1}
I20260131 03:08:30.430588 27838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.426015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56610 (local address 127.19.84.3:44133)
0131 03:08:30.426539 (+ 524us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:30.426546 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:30.426705 (+ 159us) server_negotiation.cc:408] Connection header received
0131 03:08:30.426755 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:30.426758 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:30.426775 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:30.426841 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:30.427132 (+ 291us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.428234 (+ 1102us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.429832 (+ 1598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.430396 (+ 564us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.430442 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:30.430473 (+ 31us) server_negotiation.cc:300] Negotiation successful
0131 03:08:30.430515 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":73,"threads_started":1}
I20260131 03:08:30.430609 27606 tablet_service.cc:1505] Processing CreateTablet for tablet b31be969109c4a04919e5cfa2dad702f (DEFAULT_TABLE table=test-table [id=4b244416a5f846318f318098e803c2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:08:30.430966 27606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b31be969109c4a04919e5cfa2dad702f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:30.431186 27742 tablet_service.cc:1505] Processing CreateTablet for tablet b31be969109c4a04919e5cfa2dad702f (DEFAULT_TABLE table=test-table [id=4b244416a5f846318f318098e803c2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:08:30.431490 27742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b31be969109c4a04919e5cfa2dad702f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:30.431869 27470 tablet_service.cc:1505] Processing CreateTablet for tablet b31be969109c4a04919e5cfa2dad702f (DEFAULT_TABLE table=test-table [id=4b244416a5f846318f318098e803c2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:08:30.432123 27470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b31be969109c4a04919e5cfa2dad702f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:30.434137 27839 tablet_bootstrap.cc:492] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6: Bootstrap starting.
I20260131 03:08:30.434347 27840 tablet_bootstrap.cc:492] T b31be969109c4a04919e5cfa2dad702f P e174c1a5c18148c5ac26d4824b84fe49: Bootstrap starting.
I20260131 03:08:30.434711 27841 tablet_bootstrap.cc:492] T b31be969109c4a04919e5cfa2dad702f P 0df52ee4e3d4496baaf359d610b7d34c: Bootstrap starting.
I20260131 03:08:30.435107 27839 tablet_bootstrap.cc:654] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:30.435252 27840 tablet_bootstrap.cc:654] T b31be969109c4a04919e5cfa2dad702f P e174c1a5c18148c5ac26d4824b84fe49: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:30.435352 27841 tablet_bootstrap.cc:654] T b31be969109c4a04919e5cfa2dad702f P 0df52ee4e3d4496baaf359d610b7d34c: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:30.435468 27839 log.cc:826] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:30.435600 27840 log.cc:826] T b31be969109c4a04919e5cfa2dad702f P e174c1a5c18148c5ac26d4824b84fe49: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:30.435688 27841 log.cc:826] T b31be969109c4a04919e5cfa2dad702f P 0df52ee4e3d4496baaf359d610b7d34c: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:30.436271 27839 tablet_bootstrap.cc:492] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6: No bootstrap required, opened a new log
I20260131 03:08:30.436327 27839 ts_tablet_manager.cc:1403] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260131 03:08:30.436344 27840 tablet_bootstrap.cc:492] T b31be969109c4a04919e5cfa2dad702f P e174c1a5c18148c5ac26d4824b84fe49: No bootstrap required, opened a new log
I20260131 03:08:30.436435 27840 ts_tablet_manager.cc:1403] T b31be969109c4a04919e5cfa2dad702f P e174c1a5c18148c5ac26d4824b84fe49: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:08:30.436542 27841 tablet_bootstrap.cc:492] T b31be969109c4a04919e5cfa2dad702f P 0df52ee4e3d4496baaf359d610b7d34c: No bootstrap required, opened a new log
I20260131 03:08:30.436606 27841 ts_tablet_manager.cc:1403] T b31be969109c4a04919e5cfa2dad702f P 0df52ee4e3d4496baaf359d610b7d34c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:08:30.437670 27839 raft_consensus.cc:359] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e174c1a5c18148c5ac26d4824b84fe49" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 44133 } } peers { permanent_uuid: "0df52ee4e3d4496baaf359d610b7d34c" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35541 } } peers { permanent_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39245 } }
I20260131 03:08:30.437813 27839 raft_consensus.cc:385] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:30.437847 27839 raft_consensus.cc:740] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d5b3c3b9d74484b9a5f92b2dc57fa6, State: Initialized, Role: FOLLOWER
I20260131 03:08:30.437939 27839 consensus_queue.cc:260] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e174c1a5c18148c5ac26d4824b84fe49" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 44133 } } peers { permanent_uuid: "0df52ee4e3d4496baaf359d610b7d34c" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35541 } } peers { permanent_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39245 } }
I20260131 03:08:30.437973 27840 raft_consensus.cc:359] T b31be969109c4a04919e5cfa2dad702f P e174c1a5c18148c5ac26d4824b84fe49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e174c1a5c18148c5ac26d4824b84fe49" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 44133 } } peers { permanent_uuid: "0df52ee4e3d4496baaf359d610b7d34c" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35541 } } peers { permanent_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39245 } }
I20260131 03:08:30.438117 27840 raft_consensus.cc:385] T b31be969109c4a04919e5cfa2dad702f P e174c1a5c18148c5ac26d4824b84fe49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:30.438062 27841 raft_consensus.cc:359] T b31be969109c4a04919e5cfa2dad702f P 0df52ee4e3d4496baaf359d610b7d34c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e174c1a5c18148c5ac26d4824b84fe49" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 44133 } } peers { permanent_uuid: "0df52ee4e3d4496baaf359d610b7d34c" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35541 } } peers { permanent_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39245 } }
I20260131 03:08:30.438158 27840 raft_consensus.cc:740] T b31be969109c4a04919e5cfa2dad702f P e174c1a5c18148c5ac26d4824b84fe49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e174c1a5c18148c5ac26d4824b84fe49, State: Initialized, Role: FOLLOWER
I20260131 03:08:30.438169 27841 raft_consensus.cc:385] T b31be969109c4a04919e5cfa2dad702f P 0df52ee4e3d4496baaf359d610b7d34c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:30.438198 27841 raft_consensus.cc:740] T b31be969109c4a04919e5cfa2dad702f P 0df52ee4e3d4496baaf359d610b7d34c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df52ee4e3d4496baaf359d610b7d34c, State: Initialized, Role: FOLLOWER
I20260131 03:08:30.438212 27839 ts_tablet_manager.cc:1434] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260131 03:08:30.438266 27840 consensus_queue.cc:260] T b31be969109c4a04919e5cfa2dad702f P e174c1a5c18148c5ac26d4824b84fe49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e174c1a5c18148c5ac26d4824b84fe49" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 44133 } } peers { permanent_uuid: "0df52ee4e3d4496baaf359d610b7d34c" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35541 } } peers { permanent_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39245 } }
I20260131 03:08:30.438309 27841 consensus_queue.cc:260] T b31be969109c4a04919e5cfa2dad702f P 0df52ee4e3d4496baaf359d610b7d34c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e174c1a5c18148c5ac26d4824b84fe49" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 44133 } } peers { permanent_uuid: "0df52ee4e3d4496baaf359d610b7d34c" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35541 } } peers { permanent_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39245 } }
I20260131 03:08:30.438514 27841 ts_tablet_manager.cc:1434] T b31be969109c4a04919e5cfa2dad702f P 0df52ee4e3d4496baaf359d610b7d34c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:08:30.438503 27840 ts_tablet_manager.cc:1434] T b31be969109c4a04919e5cfa2dad702f P e174c1a5c18148c5ac26d4824b84fe49: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:08:30.438597 27672 heartbeater.cc:499] Master 127.19.84.62:33805 was elected leader, sending a full tablet report...
I20260131 03:08:30.438678 27808 heartbeater.cc:499] Master 127.19.84.62:33805 was elected leader, sending a full tablet report...
I20260131 03:08:30.438810 27536 heartbeater.cc:499] Master 127.19.84.62:33805 was elected leader, sending a full tablet report...
W20260131 03:08:30.513410 27675 tablet.cc:2378] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:08:30.563155 27845 raft_consensus.cc:493] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:08:30.563282 27845 raft_consensus.cc:515] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e174c1a5c18148c5ac26d4824b84fe49" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 44133 } } peers { permanent_uuid: "0df52ee4e3d4496baaf359d610b7d34c" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35541 } } peers { permanent_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39245 } }
I20260131 03:08:30.563664 27845 leader_election.cc:290] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e174c1a5c18148c5ac26d4824b84fe49 (127.19.84.3:44133), 0df52ee4e3d4496baaf359d610b7d34c (127.19.84.1:35541)
I20260131 03:08:30.566051 27677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.563873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:35541 (local address 127.19.84.2:36505)
0131 03:08:30.563928 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:30.563939 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:30.563985 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:30.564153 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:30.564154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:30.564156 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:30.564352 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.564354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.565031 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.565032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:30.565772 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.565775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.565867 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.565872 (+ 5us) client_negotiation.cc:769] Sending connection context
0131 03:08:30.565904 (+ 32us) client_negotiation.cc:240] Negotiation successful
0131 03:08:30.565941 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260131 03:08:30.566066 27678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.563894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:44133 (local address 127.19.84.2:53525)
0131 03:08:30.563966 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:30.563978 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:30.564031 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:30.564200 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:30.564202 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:30.564203 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:30.564374 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.564376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.565035 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.565037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:30.565772 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:30.565775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.565865 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.565869 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:08:30.565902 (+ 33us) client_negotiation.cc:240] Negotiation successful
0131 03:08:30.565944 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260131 03:08:30.566475 27835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.563968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:36505 (local address 127.19.84.1:35541)
0131 03:08:30.564023 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:30.564025 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:30.564034 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:08:30.564065 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:30.564066 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:30.564081 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:30.564148 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:30.564438 (+ 290us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.564939 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.565866 (+ 927us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.566364 (+ 498us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.566372 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:30.566402 (+ 30us) server_negotiation.cc:300] Negotiation successful
0131 03:08:30.566428 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260131 03:08:30.566475 27838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.563941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:53525 (local address 127.19.84.3:44133)
0131 03:08:30.563987 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:30.563989 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:30.564023 (+ 34us) server_negotiation.cc:408] Connection header received
0131 03:08:30.564112 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:30.564113 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:30.564129 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:30.564198 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:30.564457 (+ 259us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.564945 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.565864 (+ 919us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.566345 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.566354 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:30.566376 (+ 22us) server_negotiation.cc:300] Negotiation successful
0131 03:08:30.566408 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260131 03:08:30.566906 27490 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b31be969109c4a04919e5cfa2dad702f" candidate_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0df52ee4e3d4496baaf359d610b7d34c" is_pre_election: true
I20260131 03:08:30.566943 27762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b31be969109c4a04919e5cfa2dad702f" candidate_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e174c1a5c18148c5ac26d4824b84fe49" is_pre_election: true
I20260131 03:08:30.567080 27490 raft_consensus.cc:2468] T b31be969109c4a04919e5cfa2dad702f P 0df52ee4e3d4496baaf359d610b7d34c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33d5b3c3b9d74484b9a5f92b2dc57fa6 in term 0.
I20260131 03:08:30.567126 27762 raft_consensus.cc:2468] T b31be969109c4a04919e5cfa2dad702f P e174c1a5c18148c5ac26d4824b84fe49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33d5b3c3b9d74484b9a5f92b2dc57fa6 in term 0.
I20260131 03:08:30.567358 27562 leader_election.cc:304] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0df52ee4e3d4496baaf359d610b7d34c, 33d5b3c3b9d74484b9a5f92b2dc57fa6; no voters:
I20260131 03:08:30.567663 27845 raft_consensus.cc:2804] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:08:30.567744 27845 raft_consensus.cc:493] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:08:30.567780 27845 raft_consensus.cc:3060] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:30.568486 27845 raft_consensus.cc:515] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e174c1a5c18148c5ac26d4824b84fe49" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 44133 } } peers { permanent_uuid: "0df52ee4e3d4496baaf359d610b7d34c" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35541 } } peers { permanent_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39245 } }
I20260131 03:08:30.568699 27845 leader_election.cc:290] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [CANDIDATE]: Term 1 election: Requested vote from peers e174c1a5c18148c5ac26d4824b84fe49 (127.19.84.3:44133), 0df52ee4e3d4496baaf359d610b7d34c (127.19.84.1:35541)
I20260131 03:08:30.568897 27762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b31be969109c4a04919e5cfa2dad702f" candidate_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e174c1a5c18148c5ac26d4824b84fe49"
I20260131 03:08:30.568914 27490 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b31be969109c4a04919e5cfa2dad702f" candidate_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0df52ee4e3d4496baaf359d610b7d34c"
I20260131 03:08:30.568997 27762 raft_consensus.cc:3060] T b31be969109c4a04919e5cfa2dad702f P e174c1a5c18148c5ac26d4824b84fe49 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:30.568997 27490 raft_consensus.cc:3060] T b31be969109c4a04919e5cfa2dad702f P 0df52ee4e3d4496baaf359d610b7d34c [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:30.569977 27490 raft_consensus.cc:2468] T b31be969109c4a04919e5cfa2dad702f P 0df52ee4e3d4496baaf359d610b7d34c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33d5b3c3b9d74484b9a5f92b2dc57fa6 in term 1.
I20260131 03:08:30.569977 27762 raft_consensus.cc:2468] T b31be969109c4a04919e5cfa2dad702f P e174c1a5c18148c5ac26d4824b84fe49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33d5b3c3b9d74484b9a5f92b2dc57fa6 in term 1.
I20260131 03:08:30.570313 27559 leader_election.cc:304] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d5b3c3b9d74484b9a5f92b2dc57fa6, e174c1a5c18148c5ac26d4824b84fe49; no voters:
I20260131 03:08:30.570436 27845 raft_consensus.cc:2804] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:08:30.570621 27845 raft_consensus.cc:697] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [term 1 LEADER]: Becoming Leader. State: Replica: 33d5b3c3b9d74484b9a5f92b2dc57fa6, State: Running, Role: LEADER
I20260131 03:08:30.570727 27845 consensus_queue.cc:237] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e174c1a5c18148c5ac26d4824b84fe49" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 44133 } } peers { permanent_uuid: "0df52ee4e3d4496baaf359d610b7d34c" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35541 } } peers { permanent_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39245 } }
I20260131 03:08:30.571601 26548 catalog_manager.cc:5654] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33d5b3c3b9d74484b9a5f92b2dc57fa6 (127.19.84.2). New cstate: current_term: 1 leader_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e174c1a5c18148c5ac26d4824b84fe49" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 44133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0df52ee4e3d4496baaf359d610b7d34c" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39245 } health_report { overall_health: HEALTHY } } }
I20260131 03:08:30.580927 27837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.578673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37492 (local address 127.19.84.2:39245)
0131 03:08:30.578766 (+ 93us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:30.578768 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:30.578777 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:08:30.578827 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:30.578828 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:30.578845 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:30.578915 (+ 70us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:08:30.579170 (+ 255us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.579723 (+ 553us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.580398 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.580551 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.580590 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:08:30.580682 (+ 92us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:08:30.580784 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:30.580806 (+ 22us) server_negotiation.cc:300] Negotiation successful
0131 03:08:30.580833 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260131 03:08:30.582695 27762 raft_consensus.cc:1275] T b31be969109c4a04919e5cfa2dad702f P e174c1a5c18148c5ac26d4824b84fe49 [term 1 FOLLOWER]: Refusing update from remote peer 33d5b3c3b9d74484b9a5f92b2dc57fa6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:08:30.582692 27490 raft_consensus.cc:1275] T b31be969109c4a04919e5cfa2dad702f P 0df52ee4e3d4496baaf359d610b7d34c [term 1 FOLLOWER]: Refusing update from remote peer 33d5b3c3b9d74484b9a5f92b2dc57fa6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:08:30.583081 27848 consensus_queue.cc:1048] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e174c1a5c18148c5ac26d4824b84fe49" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 44133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:08:30.583210 27845 consensus_queue.cc:1048] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0df52ee4e3d4496baaf359d610b7d34c" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:08:30.586289 27851 mvcc.cc:204] Tried to move back new op lower bound from 7249219217743724544 to 7249219217698205696. Current Snapshot: MvccSnapshot[applied={T|T < 7249219217743724544}]
I20260131 03:08:30.586700 27853 mvcc.cc:204] Tried to move back new op lower bound from 7249219217743724544 to 7249219217698205696. Current Snapshot: MvccSnapshot[applied={T|T < 7249219217743724544 or (T in {7249219217743724544})}]
I20260131 03:08:30.587688 27852 mvcc.cc:204] Tried to move back new op lower bound from 7249219217743724544 to 7249219217698205696. Current Snapshot: MvccSnapshot[applied={T|T < 7249219217743724544}]
I20260131 03:08:30.589435 26541 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:08:30.589540 26541 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:08:30.591521 26541 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 6c41b3e9bc84467e9d9758cbd2691e72: Sending DeleteTablet for 3 replicas of tablet b31be969109c4a04919e5cfa2dad702f
I20260131 03:08:30.591903 27742 tablet_service.cc:1552] Processing DeleteTablet for tablet b31be969109c4a04919e5cfa2dad702f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:30 UTC) from {username='oryx', principal='oryx/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:56610
I20260131 03:08:30.591992 27606 tablet_service.cc:1552] Processing DeleteTablet for tablet b31be969109c4a04919e5cfa2dad702f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:30 UTC) from {username='oryx', principal='oryx/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:37478
I20260131 03:08:30.592411 27862 tablet_replica.cc:333] T b31be969109c4a04919e5cfa2dad702f P e174c1a5c18148c5ac26d4824b84fe49: stopping tablet replica
I20260131 03:08:30.592504 27862 raft_consensus.cc:2243] T b31be969109c4a04919e5cfa2dad702f P e174c1a5c18148c5ac26d4824b84fe49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:08:30.592605 27862 raft_consensus.cc:2272] T b31be969109c4a04919e5cfa2dad702f P e174c1a5c18148c5ac26d4824b84fe49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:08:30.592885 27470 tablet_service.cc:1552] Processing DeleteTablet for tablet b31be969109c4a04919e5cfa2dad702f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:30 UTC) from {username='oryx', principal='oryx/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:49414
I20260131 03:08:30.593116 27862 ts_tablet_manager.cc:1916] T b31be969109c4a04919e5cfa2dad702f P e174c1a5c18148c5ac26d4824b84fe49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:08:30.593140 27864 tablet_replica.cc:333] T b31be969109c4a04919e5cfa2dad702f P 0df52ee4e3d4496baaf359d610b7d34c: stopping tablet replica
I20260131 03:08:30.593231 27864 raft_consensus.cc:2243] T b31be969109c4a04919e5cfa2dad702f P 0df52ee4e3d4496baaf359d610b7d34c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:08:30.593326 27864 raft_consensus.cc:2272] T b31be969109c4a04919e5cfa2dad702f P 0df52ee4e3d4496baaf359d610b7d34c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:08:30.593531 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 26516
I20260131 03:08:30.595683 27862 ts_tablet_manager.cc:1929] T b31be969109c4a04919e5cfa2dad702f P e174c1a5c18148c5ac26d4824b84fe49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:08:30.595764 27862 log.cc:1199] T b31be969109c4a04919e5cfa2dad702f P e174c1a5c18148c5ac26d4824b84fe49: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/b31be969109c4a04919e5cfa2dad702f
I20260131 03:08:30.596000 27862 ts_tablet_manager.cc:1950] T b31be969109c4a04919e5cfa2dad702f P e174c1a5c18148c5ac26d4824b84fe49: Deleting consensus metadata
I20260131 03:08:30.597443 27864 ts_tablet_manager.cc:1916] T b31be969109c4a04919e5cfa2dad702f P 0df52ee4e3d4496baaf359d610b7d34c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260131 03:08:30.599071 27696 connection.cc:537] server connection from 127.0.0.1:56610 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260131 03:08:30.600057 27863 tablet_replica.cc:333] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6: stopping tablet replica
I20260131 03:08:30.600205 27863 raft_consensus.cc:2243] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:08:30.600235 27864 ts_tablet_manager.cc:1929] T b31be969109c4a04919e5cfa2dad702f P 0df52ee4e3d4496baaf359d610b7d34c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:08:30.600322 27864 log.cc:1199] T b31be969109c4a04919e5cfa2dad702f P 0df52ee4e3d4496baaf359d610b7d34c: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-0/wal/wals/b31be969109c4a04919e5cfa2dad702f
I20260131 03:08:30.600535 27863 raft_consensus.cc:2272] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:08:30.600622 27864 ts_tablet_manager.cc:1950] T b31be969109c4a04919e5cfa2dad702f P 0df52ee4e3d4496baaf359d610b7d34c: Deleting consensus metadata
I20260131 03:08:30.601063 27863 ts_tablet_manager.cc:1916] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260131 03:08:30.601181 27423 connection.cc:441] server connection from 127.0.0.1:49414 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49414 (request call id 1) could send its response
I20260131 03:08:30.602890 27863 ts_tablet_manager.cc:1929] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:08:30.602965 27863 log.cc:1199] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/wal/wals/b31be969109c4a04919e5cfa2dad702f
I20260131 03:08:30.603259 27863 ts_tablet_manager.cc:1950] T b31be969109c4a04919e5cfa2dad702f P 33d5b3c3b9d74484b9a5f92b2dc57fa6: Deleting consensus metadata
W20260131 03:08:30.603619 27560 connection.cc:441] server connection from 127.0.0.1:37478 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37478 (request call id 1) could send its response
I20260131 03:08:30.609103 27820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.601077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34732 (local address 127.19.84.61:38999)
0131 03:08:30.601123 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:30.601125 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:30.601453 (+ 328us) server_negotiation.cc:408] Connection header received
0131 03:08:30.601560 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:30.601561 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:30.601572 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:30.601603 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:30.602552 (+ 949us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.603315 (+ 763us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.604204 (+ 889us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.604331 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.605862 (+ 1531us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:30.605866 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:30.605867 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:30.605874 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:30.607913 (+ 2039us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.608322 (+ 409us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.608323 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.608324 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.608355 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.608545 (+ 190us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.608546 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.608546 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.608660 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:30.608765 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:30.608897 (+ 132us) server_negotiation.cc:300] Negotiation successful
0131 03:08:30.608995 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260131 03:08:30.612674 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:30.601475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42348 (local address 127.19.84.60:36003)
0131 03:08:30.601517 (+ 42us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:30.601520 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:30.605502 (+ 3982us) server_negotiation.cc:408] Connection header received
0131 03:08:30.605664 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:30.605666 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:30.605684 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:30.605811 (+ 127us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:30.606407 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.607003 (+ 596us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:30.607707 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:30.607853 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:30.609145 (+ 1292us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:30.609150 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:30.609150 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:30.609157 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:30.611422 (+ 2265us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.611791 (+ 369us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.611792 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.611792 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.611818 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:30.612019 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:30.612020 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:30.612021 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:30.612143 (+ 122us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:30.612248 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:30.612472 (+ 224us) server_negotiation.cc:300] Negotiation successful
0131 03:08:30.612565 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260131 03:08:31.587952 27872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:31.587737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.1:41007)
0131 03:08:31.587850 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:31.587905 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
W20260131 03:08:31.588311 27536 heartbeater.cc:646] Failed to heartbeat to 127.19.84.62:33805 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
I20260131 03:08:31.588579 27873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:31.588311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.3:53551)
0131 03:08:31.588421 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:31.588524 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":55,"threads_started":1}
W20260131 03:08:31.588812 27808 heartbeater.cc:646] Failed to heartbeat to 127.19.84.62:33805 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
I20260131 03:08:31.589030 27874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:31.588756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.2:34883)
0131 03:08:31.588871 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:31.588959 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
W20260131 03:08:31.589185 27672 heartbeater.cc:646] Failed to heartbeat to 127.19.84.62:33805 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
I20260131 03:08:32.091738 27875 raft_consensus.cc:493] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6c41b3e9bc84467e9d9758cbd2691e72)
I20260131 03:08:32.091929 27875 raft_consensus.cc:515] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: false } }
I20260131 03:08:32.092232 27875 leader_election.cc:290] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c41b3e9bc84467e9d9758cbd2691e72 (127.19.84.62:33805), b75c66249f3f4d7d8a97151e41f9f403 (127.19.84.61:38999)
I20260131 03:08:32.092516 27170 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "005eb390434c4f61a808cc442be89188" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b75c66249f3f4d7d8a97151e41f9f403" is_pre_election: true
I20260131 03:08:32.092639 27876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:32.092401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.0.0.1:45644)
0131 03:08:32.092538 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:32.092592 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":66,"threads_started":1}
I20260131 03:08:32.092605 27877 raft_consensus.cc:493] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6c41b3e9bc84467e9d9758cbd2691e72)
W20260131 03:08:32.092754 27328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
I20260131 03:08:32.092922 27877 raft_consensus.cc:515] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: false } }
I20260131 03:08:32.093098 27170 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 005eb390434c4f61a808cc442be89188 in term 1.
I20260131 03:08:32.093143 27876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:32.093059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.0.0.1:45650)
0131 03:08:32.093096 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:32.093118 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260131 03:08:32.093290 27328 leader_election.cc:336] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c41b3e9bc84467e9d9758cbd2691e72 (127.19.84.62:33805): Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
I20260131 03:08:32.093357 27329 leader_election.cc:304] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 005eb390434c4f61a808cc442be89188, b75c66249f3f4d7d8a97151e41f9f403; no voters: 6c41b3e9bc84467e9d9758cbd2691e72
I20260131 03:08:32.093405 27877 leader_election.cc:290] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c41b3e9bc84467e9d9758cbd2691e72 (127.19.84.62:33805), 005eb390434c4f61a808cc442be89188 (127.19.84.60:36003)
I20260131 03:08:32.093428 27352 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b75c66249f3f4d7d8a97151e41f9f403" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "005eb390434c4f61a808cc442be89188" is_pre_election: true
I20260131 03:08:32.093488 27875 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260131 03:08:32.093536 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:32.093283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.0.0.1:45658)
0131 03:08:32.093462 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:32.093501 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":70,"threads_started":1}
I20260131 03:08:32.093530 27352 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b75c66249f3f4d7d8a97151e41f9f403 in term 1.
I20260131 03:08:32.093567 27875 raft_consensus.cc:493] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6c41b3e9bc84467e9d9758cbd2691e72)
I20260131 03:08:32.093590 27875 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 1 FOLLOWER]: Advancing to term 2
I20260131 03:08:32.093715 27145 leader_election.cc:304] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 005eb390434c4f61a808cc442be89188, b75c66249f3f4d7d8a97151e41f9f403; no voters:
W20260131 03:08:32.093719 27146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
I20260131 03:08:32.093849 27877 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260131 03:08:32.093930 27877 raft_consensus.cc:493] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6c41b3e9bc84467e9d9758cbd2691e72)
I20260131 03:08:32.093973 27877 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 1 FOLLOWER]: Advancing to term 2
I20260131 03:08:32.094102 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:32.094024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.0.0.1:45666)
0131 03:08:32.094055 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:32.094078 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260131 03:08:32.094245 27146 leader_election.cc:336] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c41b3e9bc84467e9d9758cbd2691e72 (127.19.84.62:33805): Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
I20260131 03:08:32.094550 27875 raft_consensus.cc:515] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: false } }
I20260131 03:08:32.094729 27875 leader_election.cc:290] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [CANDIDATE]: Term 2 election: Requested vote from peers 6c41b3e9bc84467e9d9758cbd2691e72 (127.19.84.62:33805), b75c66249f3f4d7d8a97151e41f9f403 (127.19.84.61:38999)
I20260131 03:08:32.094870 27877 raft_consensus.cc:515] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: false } }
I20260131 03:08:32.094923 27876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:32.094823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.0.0.1:45676)
0131 03:08:32.094870 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:32.094900 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260131 03:08:32.094913 27170 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "005eb390434c4f61a808cc442be89188" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b75c66249f3f4d7d8a97151e41f9f403"
I20260131 03:08:32.095007 27170 raft_consensus.cc:2393] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 005eb390434c4f61a808cc442be89188 in current term 2: Already voted for candidate b75c66249f3f4d7d8a97151e41f9f403 in this term.
I20260131 03:08:32.095166 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:32.095076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.0.0.1:45688)
0131 03:08:32.095116 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:32.095138 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260131 03:08:32.095204 27876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:32.095113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.0.0.1:45694)
0131 03:08:32.095149 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:32.095179 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260131 03:08:32.095285 27328 leader_election.cc:336] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6c41b3e9bc84467e9d9758cbd2691e72 (127.19.84.62:33805): Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
I20260131 03:08:32.095300 27877 leader_election.cc:290] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [CANDIDATE]: Term 2 election: Requested vote from peers 6c41b3e9bc84467e9d9758cbd2691e72 (127.19.84.62:33805), 005eb390434c4f61a808cc442be89188 (127.19.84.60:36003)
I20260131 03:08:32.095340 27328 leader_election.cc:304] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 005eb390434c4f61a808cc442be89188; no voters: 6c41b3e9bc84467e9d9758cbd2691e72, b75c66249f3f4d7d8a97151e41f9f403
I20260131 03:08:32.095485 27875 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260131 03:08:32.095490 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:32.095363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.0.0.1:45704)
0131 03:08:32.095446 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:32.095470 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260131 03:08:32.095582 27146 leader_election.cc:336] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6c41b3e9bc84467e9d9758cbd2691e72 (127.19.84.62:33805): Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
I20260131 03:08:32.095564 27352 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b75c66249f3f4d7d8a97151e41f9f403" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "005eb390434c4f61a808cc442be89188"
I20260131 03:08:32.095639 27352 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b75c66249f3f4d7d8a97151e41f9f403 in current term 2: Already voted for candidate 005eb390434c4f61a808cc442be89188 in this term.
I20260131 03:08:32.095818 27145 leader_election.cc:304] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b75c66249f3f4d7d8a97151e41f9f403; no voters: 005eb390434c4f61a808cc442be89188, 6c41b3e9bc84467e9d9758cbd2691e72
I20260131 03:08:32.095925 27877 raft_consensus.cc:2749] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260131 03:08:32.588971 27881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:32.588677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.1:52819)
0131 03:08:32.588846 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:32.588904 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":117,"threads_started":1}
I20260131 03:08:32.589387 27882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:32.589151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.3:49487)
0131 03:08:32.589271 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:32.589306 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
I20260131 03:08:32.589859 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:32.589533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.2:49719)
0131 03:08:32.589759 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:32.589794 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":68,"threads_started":1}
I20260131 03:08:33.589938 27884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.589676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.1:47659)
0131 03:08:33.589837 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.589873 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":95,"threads_started":1}
I20260131 03:08:33.590490 27886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.590288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.2:56545)
0131 03:08:33.590397 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.590424 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":74,"threads_started":1}
I20260131 03:08:33.590679 27885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.590123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.3:33079)
0131 03:08:33.590530 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.590631 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":52,"threads_started":1}
I20260131 03:08:33.620957 27887 raft_consensus.cc:493] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:08:33.621079 27887 raft_consensus.cc:515] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: false } }
I20260131 03:08:33.621290 27887 leader_election.cc:290] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6c41b3e9bc84467e9d9758cbd2691e72 (127.19.84.62:33805), 005eb390434c4f61a808cc442be89188 (127.19.84.60:36003)
I20260131 03:08:33.621626 27352 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b75c66249f3f4d7d8a97151e41f9f403" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "005eb390434c4f61a808cc442be89188" is_pre_election: true
I20260131 03:08:33.621659 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.621476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.0.0.1:45936)
0131 03:08:33.621588 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.621621 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":60,"threads_started":1}
I20260131 03:08:33.621726 27352 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b75c66249f3f4d7d8a97151e41f9f403 in term 2.
I20260131 03:08:33.621912 27145 leader_election.cc:304] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 005eb390434c4f61a808cc442be89188, b75c66249f3f4d7d8a97151e41f9f403; no voters:
I20260131 03:08:33.622015 27887 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260131 03:08:33.622061 27887 raft_consensus.cc:493] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:08:33.622089 27887 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 2 FOLLOWER]: Advancing to term 3
I20260131 03:08:33.622098 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.622006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.0.0.1:45952)
0131 03:08:33.622039 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.622074 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260131 03:08:33.622185 27146 leader_election.cc:336] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6c41b3e9bc84467e9d9758cbd2691e72 (127.19.84.62:33805): Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
I20260131 03:08:33.622861 27887 raft_consensus.cc:515] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: false } }
I20260131 03:08:33.623010 27887 leader_election.cc:290] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [CANDIDATE]: Term 3 election: Requested vote from peers 6c41b3e9bc84467e9d9758cbd2691e72 (127.19.84.62:33805), 005eb390434c4f61a808cc442be89188 (127.19.84.60:36003)
I20260131 03:08:33.623203 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.623098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.0.0.1:45968)
0131 03:08:33.623139 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.623183 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260131 03:08:33.623202 27352 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b75c66249f3f4d7d8a97151e41f9f403" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "005eb390434c4f61a808cc442be89188"
I20260131 03:08:33.623275 27352 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 2 FOLLOWER]: Advancing to term 3
I20260131 03:08:33.623579 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.623492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.0.0.1:45972)
0131 03:08:33.623535 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.623559 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260131 03:08:33.623683 27146 leader_election.cc:336] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6c41b3e9bc84467e9d9758cbd2691e72 (127.19.84.62:33805): Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
I20260131 03:08:33.624089 27352 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b75c66249f3f4d7d8a97151e41f9f403 in term 3.
I20260131 03:08:33.624246 27145 leader_election.cc:304] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 005eb390434c4f61a808cc442be89188, b75c66249f3f4d7d8a97151e41f9f403; no voters: 6c41b3e9bc84467e9d9758cbd2691e72
I20260131 03:08:33.624331 27887 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 3 FOLLOWER]: Leader election won for term 3
I20260131 03:08:33.624449 27887 raft_consensus.cc:697] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [term 3 LEADER]: Becoming Leader. State: Replica: b75c66249f3f4d7d8a97151e41f9f403, State: Running, Role: LEADER
I20260131 03:08:33.624518 27887 consensus_queue.cc:237] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: false } }
I20260131 03:08:33.624881 27890 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b75c66249f3f4d7d8a97151e41f9f403. Latest consensus state: current_term: 3 leader_uuid: "b75c66249f3f4d7d8a97151e41f9f403" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: false } } }
I20260131 03:08:33.624948 27890 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:33.625138 27892 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:08:33.625528 27892 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:08:33.625591 27892 catalog_manager.cc:1269] Loaded cluster ID: 00b8836497954b1f8f0752703e43dd26
I20260131 03:08:33.625608 27892 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:08:33.625792 27892 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:08:33.625875 27892 catalog_manager.cc:6038] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403: Loaded TSK: 0
I20260131 03:08:33.626044 27892 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:08:33.674497 27159 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34732:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:08:33.675817 27159 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:08:33.676873 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.676727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.0.0.1:45994)
0131 03:08:33.676801 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.676837 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260131 03:08:33.676891 27352 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [term 3 FOLLOWER]: Refusing update from remote peer b75c66249f3f4d7d8a97151e41f9f403: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260131 03:08:33.677168 27887 consensus_queue.cc:1048] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260131 03:08:33.677323 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.677200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.0.0.1:45998)
0131 03:08:33.677244 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.677293 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260131 03:08:33.677448 27146 consensus_peers.cc:597] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 -> Peer 6c41b3e9bc84467e9d9758cbd2691e72 (127.19.84.62:33805): Couldn't send request to peer 6c41b3e9bc84467e9d9758cbd2691e72. Status: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260131 03:08:33.677829 27894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b75c66249f3f4d7d8a97151e41f9f403. Latest consensus state: current_term: 3 leader_uuid: "b75c66249f3f4d7d8a97151e41f9f403" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: false } } }
I20260131 03:08:33.677917 27894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:08:33.678220 27893 mvcc.cc:204] Tried to move back new op lower bound from 7249219230417993728 to 7249219230206730240. Current Snapshot: MvccSnapshot[applied={T|T < 7249219230417993728}]
I20260131 03:08:33.678339 27887 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b75c66249f3f4d7d8a97151e41f9f403" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: false } } }
I20260131 03:08:33.678442 27887 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:33.678707 27894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b75c66249f3f4d7d8a97151e41f9f403" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6c41b3e9bc84467e9d9758cbd2691e72" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 33805 } } peers { permanent_uuid: "b75c66249f3f4d7d8a97151e41f9f403" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 38999 } attrs { promote: false } } peers { permanent_uuid: "005eb390434c4f61a808cc442be89188" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 36003 } attrs { promote: false } } }
I20260131 03:08:33.678804 27894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 005eb390434c4f61a808cc442be89188 [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:08:33.678922 27895 mvcc.cc:204] Tried to move back new op lower bound from 7249219230417993728 to 7249219230206730240. Current Snapshot: MvccSnapshot[applied={T|T < 7249219230417993728 or (T in {7249219230417993728})}]
I20260131 03:08:33.682768 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.680029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:35541 (local address 127.0.0.1:47884)
0131 03:08:33.680118 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.680128 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:33.680176 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:33.680569 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:33.680570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:33.680571 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:33.680762 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:33.680763 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:33.681651 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:33.681652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:33.682518 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:33.682520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:33.682593 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:33.682595 (+ 2us) client_negotiation.cc:769] Sending connection context
0131 03:08:33.682631 (+ 36us) client_negotiation.cc:240] Negotiation successful
0131 03:08:33.682672 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260131 03:08:33.682874 27900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.680038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:44133 (local address 127.0.0.1:33806)
0131 03:08:33.680274 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.680285 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:33.680330 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:33.680635 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:33.680636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:33.680639 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:33.680846 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:33.680848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:33.681574 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:33.681575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:33.682518 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:33.682520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:33.682591 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:33.682596 (+ 5us) client_negotiation.cc:769] Sending connection context
0131 03:08:33.682629 (+ 33us) client_negotiation.cc:240] Negotiation successful
0131 03:08:33.682666 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":81,"threads_started":1}
I20260131 03:08:33.683054 27901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.680063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:39245 (local address 127.0.0.1:34024)
0131 03:08:33.680438 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.680448 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:33.680489 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:33.680796 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:33.680797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:33.680799 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:33.680966 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:33.680967 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:33.681819 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:33.681819 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:33.682536 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:33.682538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:33.682621 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:33.682625 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:08:33.682657 (+ 32us) client_negotiation.cc:240] Negotiation successful
0131 03:08:33.682696 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":70,"threads_started":1}
I20260131 03:08:33.683279 27902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.680180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33806 (local address 127.19.84.3:44133)
0131 03:08:33.680442 (+ 262us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:33.680444 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:33.680461 (+ 17us) server_negotiation.cc:408] Connection header received
0131 03:08:33.680498 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:33.680499 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:33.680512 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:33.680573 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:33.680926 (+ 353us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:33.681472 (+ 546us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:33.682628 (+ 1156us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:33.683134 (+ 506us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:33.683144 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:33.683169 (+ 25us) server_negotiation.cc:300] Negotiation successful
0131 03:08:33.683208 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":43,"threads_started":1}
I20260131 03:08:33.683418 27904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.680307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34024 (local address 127.19.84.2:39245)
0131 03:08:33.680625 (+ 318us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:33.680627 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:33.680644 (+ 17us) server_negotiation.cc:408] Connection header received
0131 03:08:33.680669 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:33.680670 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:33.680680 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:33.680749 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:33.681050 (+ 301us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:33.681725 (+ 675us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:33.682786 (+ 1061us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:33.683244 (+ 458us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:33.683251 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:33.683270 (+ 19us) server_negotiation.cc:300] Negotiation successful
0131 03:08:33.683310 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":57,"threads_started":1}
I20260131 03:08:33.683512 27742 tablet_service.cc:1505] Processing CreateTablet for tablet 2ad61be8122b4b35bb4557ba0583e4f5 (DEFAULT_TABLE table=test-table [id=82ed724b36db4bfeaa247e4b0794b001]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:08:33.683655 27742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ad61be8122b4b35bb4557ba0583e4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:33.683705 27606 tablet_service.cc:1505] Processing CreateTablet for tablet 2ad61be8122b4b35bb4557ba0583e4f5 (DEFAULT_TABLE table=test-table [id=82ed724b36db4bfeaa247e4b0794b001]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:08:33.683835 27606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ad61be8122b4b35bb4557ba0583e4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:33.685168 27906 tablet_bootstrap.cc:492] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49: Bootstrap starting.
I20260131 03:08:33.685593 27906 tablet_bootstrap.cc:654] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:33.685606 27903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.680216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47884 (local address 127.19.84.1:35541)
0131 03:08:33.680431 (+ 215us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:33.680434 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:33.680451 (+ 17us) server_negotiation.cc:408] Connection header received
0131 03:08:33.680481 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:33.680482 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:33.680508 (+ 26us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:33.680569 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:33.680841 (+ 272us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:33.681578 (+ 737us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:33.683285 (+ 1707us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:33.685485 (+ 2200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:33.685493 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:33.685516 (+ 23us) server_negotiation.cc:300] Negotiation successful
0131 03:08:33.685544 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":56,"threads_started":1}
I20260131 03:08:33.685847 27470 tablet_service.cc:1505] Processing CreateTablet for tablet 2ad61be8122b4b35bb4557ba0583e4f5 (DEFAULT_TABLE table=test-table [id=82ed724b36db4bfeaa247e4b0794b001]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:08:33.685966 27470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ad61be8122b4b35bb4557ba0583e4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:33.685992 27905 tablet_bootstrap.cc:492] T 2ad61be8122b4b35bb4557ba0583e4f5 P 33d5b3c3b9d74484b9a5f92b2dc57fa6: Bootstrap starting.
I20260131 03:08:33.686340 27906 tablet_bootstrap.cc:492] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49: No bootstrap required, opened a new log
I20260131 03:08:33.686403 27906 ts_tablet_manager.cc:1403] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:33.686420 27905 tablet_bootstrap.cc:654] T 2ad61be8122b4b35bb4557ba0583e4f5 P 33d5b3c3b9d74484b9a5f92b2dc57fa6: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:33.686635 27906 raft_consensus.cc:359] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df52ee4e3d4496baaf359d610b7d34c" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35541 } } peers { permanent_uuid: "e174c1a5c18148c5ac26d4824b84fe49" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 44133 } } peers { permanent_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39245 } }
I20260131 03:08:33.686749 27906 raft_consensus.cc:385] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:33.686796 27906 raft_consensus.cc:740] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e174c1a5c18148c5ac26d4824b84fe49, State: Initialized, Role: FOLLOWER
I20260131 03:08:33.686885 27906 consensus_queue.cc:260] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df52ee4e3d4496baaf359d610b7d34c" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35541 } } peers { permanent_uuid: "e174c1a5c18148c5ac26d4824b84fe49" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 44133 } } peers { permanent_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39245 } }
I20260131 03:08:33.687060 27905 tablet_bootstrap.cc:492] T 2ad61be8122b4b35bb4557ba0583e4f5 P 33d5b3c3b9d74484b9a5f92b2dc57fa6: No bootstrap required, opened a new log
I20260131 03:08:33.687077 27906 ts_tablet_manager.cc:1434] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:33.687117 27905 ts_tablet_manager.cc:1403] T 2ad61be8122b4b35bb4557ba0583e4f5 P 33d5b3c3b9d74484b9a5f92b2dc57fa6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:33.687280 27905 raft_consensus.cc:359] T 2ad61be8122b4b35bb4557ba0583e4f5 P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df52ee4e3d4496baaf359d610b7d34c" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35541 } } peers { permanent_uuid: "e174c1a5c18148c5ac26d4824b84fe49" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 44133 } } peers { permanent_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39245 } }
I20260131 03:08:33.687443 27905 raft_consensus.cc:385] T 2ad61be8122b4b35bb4557ba0583e4f5 P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:33.687515 27905 raft_consensus.cc:740] T 2ad61be8122b4b35bb4557ba0583e4f5 P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d5b3c3b9d74484b9a5f92b2dc57fa6, State: Initialized, Role: FOLLOWER
I20260131 03:08:33.687613 27885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.687298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.3:58559)
0131 03:08:33.687554 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.687586 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236}
I20260131 03:08:33.687659 27910 tablet_bootstrap.cc:492] T 2ad61be8122b4b35bb4557ba0583e4f5 P 0df52ee4e3d4496baaf359d610b7d34c: Bootstrap starting.
I20260131 03:08:33.687638 27905 consensus_queue.cc:260] T 2ad61be8122b4b35bb4557ba0583e4f5 P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df52ee4e3d4496baaf359d610b7d34c" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35541 } } peers { permanent_uuid: "e174c1a5c18148c5ac26d4824b84fe49" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 44133 } } peers { permanent_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39245 } }
I20260131 03:08:33.687839 27905 ts_tablet_manager.cc:1434] T 2ad61be8122b4b35bb4557ba0583e4f5 P 33d5b3c3b9d74484b9a5f92b2dc57fa6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260131 03:08:33.688082 27886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.687966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.2:35151)
0131 03:08:33.688022 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.688047 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260131 03:08:33.688127 27910 tablet_bootstrap.cc:654] T 2ad61be8122b4b35bb4557ba0583e4f5 P 0df52ee4e3d4496baaf359d610b7d34c: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:33.688354 27886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.688282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.2:53835)
0131 03:08:33.688312 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.688330 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260131 03:08:33.688719 27885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.688635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.3:38849)
0131 03:08:33.688674 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.688696 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260131 03:08:33.688877 27674 heartbeater.cc:499] Master 127.19.84.61:38999 was elected leader, sending a full tablet report...
I20260131 03:08:33.689127 27910 tablet_bootstrap.cc:492] T 2ad61be8122b4b35bb4557ba0583e4f5 P 0df52ee4e3d4496baaf359d610b7d34c: No bootstrap required, opened a new log
I20260131 03:08:33.689131 27810 heartbeater.cc:499] Master 127.19.84.61:38999 was elected leader, sending a full tablet report...
I20260131 03:08:33.689218 27910 ts_tablet_manager.cc:1403] T 2ad61be8122b4b35bb4557ba0583e4f5 P 0df52ee4e3d4496baaf359d610b7d34c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:08:33.689361 27910 raft_consensus.cc:359] T 2ad61be8122b4b35bb4557ba0583e4f5 P 0df52ee4e3d4496baaf359d610b7d34c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df52ee4e3d4496baaf359d610b7d34c" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35541 } } peers { permanent_uuid: "e174c1a5c18148c5ac26d4824b84fe49" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 44133 } } peers { permanent_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39245 } }
I20260131 03:08:33.689465 27910 raft_consensus.cc:385] T 2ad61be8122b4b35bb4557ba0583e4f5 P 0df52ee4e3d4496baaf359d610b7d34c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:33.689507 27910 raft_consensus.cc:740] T 2ad61be8122b4b35bb4557ba0583e4f5 P 0df52ee4e3d4496baaf359d610b7d34c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df52ee4e3d4496baaf359d610b7d34c, State: Initialized, Role: FOLLOWER
I20260131 03:08:33.689585 27910 consensus_queue.cc:260] T 2ad61be8122b4b35bb4557ba0583e4f5 P 0df52ee4e3d4496baaf359d610b7d34c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df52ee4e3d4496baaf359d610b7d34c" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35541 } } peers { permanent_uuid: "e174c1a5c18148c5ac26d4824b84fe49" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 44133 } } peers { permanent_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39245 } }
I20260131 03:08:33.689738 27910 ts_tablet_manager.cc:1434] T 2ad61be8122b4b35bb4557ba0583e4f5 P 0df52ee4e3d4496baaf359d610b7d34c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:33.690024 27884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.689904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.1:40457)
0131 03:08:33.689963 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.689992 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260131 03:08:33.690353 27884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.690248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.1:40155)
0131 03:08:33.690298 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.690325 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260131 03:08:33.690403 27538 heartbeater.cc:499] Master 127.19.84.61:38999 was elected leader, sending a full tablet report...
I20260131 03:08:33.726032 27909 raft_consensus.cc:493] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:08:33.726127 27909 raft_consensus.cc:515] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df52ee4e3d4496baaf359d610b7d34c" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35541 } } peers { permanent_uuid: "e174c1a5c18148c5ac26d4824b84fe49" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 44133 } } peers { permanent_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39245 } }
I20260131 03:08:33.726473 27909 leader_election.cc:290] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0df52ee4e3d4496baaf359d610b7d34c (127.19.84.1:35541), 33d5b3c3b9d74484b9a5f92b2dc57fa6 (127.19.84.2:39245)
I20260131 03:08:33.729470 27885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.726615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:39245 (local address 127.19.84.3:33953)
0131 03:08:33.726710 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.726723 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:33.726802 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:33.727006 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:33.727007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:33.727009 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:33.727274 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:33.727275 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:33.728271 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:33.728272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:33.729243 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:33.729245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:33.729327 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:33.729329 (+ 2us) client_negotiation.cc:769] Sending connection context
0131 03:08:33.729363 (+ 34us) client_negotiation.cc:240] Negotiation successful
0131 03:08:33.729394 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260131 03:08:33.729511 27914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.726631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:35541 (local address 127.19.84.3:35853)
0131 03:08:33.726797 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.726806 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:33.726864 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:33.727006 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:33.727007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:33.727010 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:33.727274 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:33.727275 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:33.728291 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:33.728292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:33.729212 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:33.729215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:33.729296 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:33.729302 (+ 6us) client_negotiation.cc:769] Sending connection context
0131 03:08:33.729339 (+ 37us) client_negotiation.cc:240] Negotiation successful
0131 03:08:33.729381 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":103,"threads_started":1}
I20260131 03:08:33.729974 27904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.726731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:33953 (local address 127.19.84.2:39245)
0131 03:08:33.726784 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:33.726786 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:33.726797 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:08:33.726885 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:33.726886 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:33.726896 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:33.726965 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:33.727398 (+ 433us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:33.728178 (+ 780us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:33.729340 (+ 1162us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:33.729838 (+ 498us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:33.729847 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:33.729873 (+ 26us) server_negotiation.cc:300] Negotiation successful
0131 03:08:33.729905 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260131 03:08:33.729976 27903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.726691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:35853 (local address 127.19.84.1:35541)
0131 03:08:33.726739 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:33.726741 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:33.726887 (+ 146us) server_negotiation.cc:408] Connection header received
0131 03:08:33.726917 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:33.726917 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:33.726927 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:33.726979 (+ 52us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:33.727398 (+ 419us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:33.728171 (+ 773us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:33.729340 (+ 1169us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:33.729839 (+ 499us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:33.729847 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:33.729873 (+ 26us) server_negotiation.cc:300] Negotiation successful
0131 03:08:33.729908 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260131 03:08:33.730219 27490 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ad61be8122b4b35bb4557ba0583e4f5" candidate_uuid: "e174c1a5c18148c5ac26d4824b84fe49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0df52ee4e3d4496baaf359d610b7d34c" is_pre_election: true
I20260131 03:08:33.730293 27490 raft_consensus.cc:2468] T 2ad61be8122b4b35bb4557ba0583e4f5 P 0df52ee4e3d4496baaf359d610b7d34c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e174c1a5c18148c5ac26d4824b84fe49 in term 0.
I20260131 03:08:33.730242 27626 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ad61be8122b4b35bb4557ba0583e4f5" candidate_uuid: "e174c1a5c18148c5ac26d4824b84fe49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" is_pre_election: true
I20260131 03:08:33.730410 27626 raft_consensus.cc:2468] T 2ad61be8122b4b35bb4557ba0583e4f5 P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e174c1a5c18148c5ac26d4824b84fe49 in term 0.
I20260131 03:08:33.730509 27698 leader_election.cc:304] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0df52ee4e3d4496baaf359d610b7d34c, e174c1a5c18148c5ac26d4824b84fe49; no voters:
I20260131 03:08:33.730665 27909 raft_consensus.cc:2804] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:08:33.730764 27909 raft_consensus.cc:493] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:08:33.730799 27909 raft_consensus.cc:3060] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:33.732851 27909 raft_consensus.cc:515] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df52ee4e3d4496baaf359d610b7d34c" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35541 } } peers { permanent_uuid: "e174c1a5c18148c5ac26d4824b84fe49" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 44133 } } peers { permanent_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39245 } }
I20260131 03:08:33.733062 27909 leader_election.cc:290] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49 [CANDIDATE]: Term 1 election: Requested vote from peers 0df52ee4e3d4496baaf359d610b7d34c (127.19.84.1:35541), 33d5b3c3b9d74484b9a5f92b2dc57fa6 (127.19.84.2:39245)
I20260131 03:08:33.733246 27490 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ad61be8122b4b35bb4557ba0583e4f5" candidate_uuid: "e174c1a5c18148c5ac26d4824b84fe49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0df52ee4e3d4496baaf359d610b7d34c"
I20260131 03:08:33.733345 27490 raft_consensus.cc:3060] T 2ad61be8122b4b35bb4557ba0583e4f5 P 0df52ee4e3d4496baaf359d610b7d34c [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:33.733439 27626 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2ad61be8122b4b35bb4557ba0583e4f5" candidate_uuid: "e174c1a5c18148c5ac26d4824b84fe49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6"
I20260131 03:08:33.733505 27626 raft_consensus.cc:3060] T 2ad61be8122b4b35bb4557ba0583e4f5 P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:33.734092 27490 raft_consensus.cc:2468] T 2ad61be8122b4b35bb4557ba0583e4f5 P 0df52ee4e3d4496baaf359d610b7d34c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e174c1a5c18148c5ac26d4824b84fe49 in term 1.
I20260131 03:08:33.734288 27698 leader_election.cc:304] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0df52ee4e3d4496baaf359d610b7d34c, e174c1a5c18148c5ac26d4824b84fe49; no voters:
I20260131 03:08:33.734387 27909 raft_consensus.cc:2804] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:08:33.734382 27626 raft_consensus.cc:2468] T 2ad61be8122b4b35bb4557ba0583e4f5 P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e174c1a5c18148c5ac26d4824b84fe49 in term 1.
I20260131 03:08:33.734549 27909 raft_consensus.cc:697] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49 [term 1 LEADER]: Becoming Leader. State: Replica: e174c1a5c18148c5ac26d4824b84fe49, State: Running, Role: LEADER
I20260131 03:08:33.734678 27909 consensus_queue.cc:237] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df52ee4e3d4496baaf359d610b7d34c" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35541 } } peers { permanent_uuid: "e174c1a5c18148c5ac26d4824b84fe49" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 44133 } } peers { permanent_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39245 } }
I20260131 03:08:33.735049 27914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.734928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.3:53457)
0131 03:08:33.734982 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.735015 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260131 03:08:33.735479 27159 catalog_manager.cc:5654] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49 reported cstate change: term changed from 0 to 1, leader changed from <none> to e174c1a5c18148c5ac26d4824b84fe49 (127.19.84.3). New cstate: current_term: 1 leader_uuid: "e174c1a5c18148c5ac26d4824b84fe49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df52ee4e3d4496baaf359d610b7d34c" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e174c1a5c18148c5ac26d4824b84fe49" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 44133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39245 } health_report { overall_health: UNKNOWN } } }
I20260131 03:08:33.740763 27902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.738742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33822 (local address 127.19.84.3:44133)
0131 03:08:33.738798 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:33.738800 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:33.738808 (+ 8us) server_negotiation.cc:408] Connection header received
0131 03:08:33.738865 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:33.738865 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:33.738880 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:33.738929 (+ 49us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:08:33.739142 (+ 213us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:33.739665 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:33.740198 (+ 533us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:33.740355 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:33.740386 (+ 31us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:08:33.740493 (+ 107us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:08:33.740552 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:33.740631 (+ 79us) server_negotiation.cc:300] Negotiation successful
0131 03:08:33.740652 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260131 03:08:33.741978 27490 raft_consensus.cc:1275] T 2ad61be8122b4b35bb4557ba0583e4f5 P 0df52ee4e3d4496baaf359d610b7d34c [term 1 FOLLOWER]: Refusing update from remote peer e174c1a5c18148c5ac26d4824b84fe49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:08:33.742076 27626 raft_consensus.cc:1275] T 2ad61be8122b4b35bb4557ba0583e4f5 P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [term 1 FOLLOWER]: Refusing update from remote peer e174c1a5c18148c5ac26d4824b84fe49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:08:33.742215 27915 consensus_queue.cc:1048] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0df52ee4e3d4496baaf359d610b7d34c" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:08:33.742367 27909 consensus_queue.cc:1048] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d5b3c3b9d74484b9a5f92b2dc57fa6" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:08:33.742913 27884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.742795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.1:50555)
0131 03:08:33.742830 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.742861 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260131 03:08:33.743448 27886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.743317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.2:51987)
0131 03:08:33.743358 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.743412 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260131 03:08:33.744550 27914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.744368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.3:52799)
0131 03:08:33.744490 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.744519 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260131 03:08:33.744630 27884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.744543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.1:36427)
0131 03:08:33.744582 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.744604 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260131 03:08:33.745644 27914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.745544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.3:51139)
0131 03:08:33.745587 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.745617 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260131 03:08:33.745954 27886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.745860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:33805 (local address 127.19.84.2:37733)
0131 03:08:33.745904 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.745927 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:33805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260131 03:08:33.746671 27159 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:08:33.746798 27159 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:08:33.748143 27159 catalog_manager.cc:5941] T 00000000000000000000000000000000 P b75c66249f3f4d7d8a97151e41f9f403: Sending DeleteTablet for 3 replicas of tablet 2ad61be8122b4b35bb4557ba0583e4f5
I20260131 03:08:33.748399 27470 tablet_service.cc:1552] Processing DeleteTablet for tablet 2ad61be8122b4b35bb4557ba0583e4f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:33 UTC) from {username='oryx', principal='oryx/127.19.84.61@KRBTEST.COM'} at 127.0.0.1:47884
I20260131 03:08:33.748569 27742 tablet_service.cc:1552] Processing DeleteTablet for tablet 2ad61be8122b4b35bb4557ba0583e4f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:33 UTC) from {username='oryx', principal='oryx/127.19.84.61@KRBTEST.COM'} at 127.0.0.1:33806
I20260131 03:08:33.748719 27606 tablet_service.cc:1552] Processing DeleteTablet for tablet 2ad61be8122b4b35bb4557ba0583e4f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:33 UTC) from {username='oryx', principal='oryx/127.19.84.61@KRBTEST.COM'} at 127.0.0.1:34024
I20260131 03:08:33.748759 27928 tablet_replica.cc:333] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49: stopping tablet replica
I20260131 03:08:33.748818 27928 raft_consensus.cc:2243] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:08:33.748898 27928 raft_consensus.cc:2272] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:08:33.748951 27927 tablet_replica.cc:333] T 2ad61be8122b4b35bb4557ba0583e4f5 P 0df52ee4e3d4496baaf359d610b7d34c: stopping tablet replica
I20260131 03:08:33.748996 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 27406
I20260131 03:08:33.749761 27928 ts_tablet_manager.cc:1916] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:08:33.751910 27928 ts_tablet_manager.cc:1929] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:08:33.752044 27928 log.cc:1199] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/2ad61be8122b4b35bb4557ba0583e4f5
I20260131 03:08:33.752384 27928 ts_tablet_manager.cc:1950] T 2ad61be8122b4b35bb4557ba0583e4f5 P e174c1a5c18148c5ac26d4824b84fe49: Deleting consensus metadata
I20260131 03:08:33.753294 27144 catalog_manager.cc:4985] TS e174c1a5c18148c5ac26d4824b84fe49 (127.19.84.3:44133): tablet 2ad61be8122b4b35bb4557ba0583e4f5 (table test-table [id=82ed724b36db4bfeaa247e4b0794b001]) successfully deleted
I20260131 03:08:33.753391 27929 tablet_replica.cc:333] T 2ad61be8122b4b35bb4557ba0583e4f5 P 33d5b3c3b9d74484b9a5f92b2dc57fa6: stopping tablet replica
I20260131 03:08:33.753535 27929 raft_consensus.cc:2243] T 2ad61be8122b4b35bb4557ba0583e4f5 P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:08:33.753602 27929 raft_consensus.cc:2272] T 2ad61be8122b4b35bb4557ba0583e4f5 P 33d5b3c3b9d74484b9a5f92b2dc57fa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:08:33.753978 27929 ts_tablet_manager.cc:1916] T 2ad61be8122b4b35bb4557ba0583e4f5 P 33d5b3c3b9d74484b9a5f92b2dc57fa6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:08:33.755571 27929 ts_tablet_manager.cc:1929] T 2ad61be8122b4b35bb4557ba0583e4f5 P 33d5b3c3b9d74484b9a5f92b2dc57fa6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:08:33.755635 27929 log.cc:1199] T 2ad61be8122b4b35bb4557ba0583e4f5 P 33d5b3c3b9d74484b9a5f92b2dc57fa6: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769828849314684-19792-0/minicluster-data/ts-1/wal/wals/2ad61be8122b4b35bb4557ba0583e4f5
I20260131 03:08:33.755877 27929 ts_tablet_manager.cc:1950] T 2ad61be8122b4b35bb4557ba0583e4f5 P 33d5b3c3b9d74484b9a5f92b2dc57fa6: Deleting consensus metadata
I20260131 03:08:33.756454 27146 catalog_manager.cc:4985] TS 33d5b3c3b9d74484b9a5f92b2dc57fa6 (127.19.84.2:39245): tablet 2ad61be8122b4b35bb4557ba0583e4f5 (table test-table [id=82ed724b36db4bfeaa247e4b0794b001]) successfully deleted
I20260131 03:08:33.757107 27901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:33.756994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:35541 (local address 127.0.0.1:47892)
0131 03:08:33.757038 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:33.757067 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.1:35541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260131 03:08:33.757251 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 27543
W20260131 03:08:33.758258 27147 catalog_manager.cc:4712] TS 0df52ee4e3d4496baaf359d610b7d34c (127.19.84.1:35541): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ad61be8122b4b35bb4557ba0583e4f5: Network error: Client connection negotiation failed: client connection to 127.19.84.1:35541: connect: Connection refused (error 111)
I20260131 03:08:33.763197 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 27679
I20260131 03:08:33.768884 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 27020
I20260131 03:08:33.773869 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 27143
2026-01-31T03:08:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8251 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:08:33 dist-test-slave-5nnp krb5kdc[27932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:08:33 dist-test-slave-5nnp krb5kdc[27932](info): set up 2 sockets
Jan 31 03:08:33 dist-test-slave-5nnp krb5kdc[27932](info): commencing operation
krb5kdc: starting...
W20260131 03:08:35.800511 19792 mini_kdc.cc:121] Time spent starting KDC: real 2.004s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:08:35 dist-test-slave-5nnp krb5kdc[27932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:08:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:08:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/oryx.keytab.
I20260131 03:08:35.943148 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:41959
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:41293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:41959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:36.026135 27948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:36.026396 27948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:36.026437 27948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:36.029098 27948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:08:36.029175 27948 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260131 03:08:36.029201 27948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:36.029237 27948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:36.029259 27948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:08:36.029289 27948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:08:36.031770 27948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:41293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:41959
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:41959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:36.032055 27948 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:36.032274 27948 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260131 03:08:36.035920 27948 server_base.cc:1060] running on GCE node
W20260131 03:08:36.035939 27954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:36.035884 27953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:36.035884 27956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:36.036293 27948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:36.036566 27948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:36.037720 27948 hybrid_clock.cc:648] HybridClock initialized: now 1769828916037683 us; error 55 us; skew 500 ppm
Jan 31 03:08:36 dist-test-slave-5nnp krb5kdc[27932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828916, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:36.040582 27948 init.cc:377] Logged in from keytab as oryx/127.19.84.62@KRBTEST.COM (short username oryx)
I20260131 03:08:36.041177 27948 webserver.cc:492] Webserver started at http://127.19.84.62:39561/ using document root <none> and password file <none>
I20260131 03:08:36.041392 27948 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:36.041452 27948 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:36.041719 27948 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:36.042726 27948 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "8bd0963c2d344f058b9aa4fd3fc2cce1"
format_stamp: "Formatted at 2026-01-31 03:08:36 on dist-test-slave-5nnp"
server_key: "69eb39a40e1f2cd5dbe3db831248346f"
server_key_iv: "e16f1818d0372000fe90dc6e1f018682"
server_key_version: "encryptionkey@0"
I20260131 03:08:36.043054 27948 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "8bd0963c2d344f058b9aa4fd3fc2cce1"
format_stamp: "Formatted at 2026-01-31 03:08:36 on dist-test-slave-5nnp"
server_key: "69eb39a40e1f2cd5dbe3db831248346f"
server_key_iv: "e16f1818d0372000fe90dc6e1f018682"
server_key_version: "encryptionkey@0"
I20260131 03:08:36.044386 27948 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260131 03:08:36.045225 27963 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:36.045420 27948 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:08:36.045531 27948 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "8bd0963c2d344f058b9aa4fd3fc2cce1"
format_stamp: "Formatted at 2026-01-31 03:08:36 on dist-test-slave-5nnp"
server_key: "69eb39a40e1f2cd5dbe3db831248346f"
server_key_iv: "e16f1818d0372000fe90dc6e1f018682"
server_key_version: "encryptionkey@0"
I20260131 03:08:36.045614 27948 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:36.061852 27948 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:36.063958 27948 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:36.064124 27948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:36.069514 27948 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:41959
I20260131 03:08:36.069578 28025 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:41959 every 8 connection(s)
I20260131 03:08:36.069905 27948 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:08:36.070793 28026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:36.072979 28026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1: Bootstrap starting.
I20260131 03:08:36.073513 28026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:36.073724 28026 log.cc:826] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:36.074450 28026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1: No bootstrap required, opened a new log
I20260131 03:08:36.075737 28026 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd0963c2d344f058b9aa4fd3fc2cce1" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41959 } }
I20260131 03:08:36.075870 28026 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:36.075891 28026 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bd0963c2d344f058b9aa4fd3fc2cce1, State: Initialized, Role: FOLLOWER
I20260131 03:08:36.076021 28026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd0963c2d344f058b9aa4fd3fc2cce1" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41959 } }
I20260131 03:08:36.076097 28026 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:08:36.076120 28026 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:08:36.076148 28026 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:36.076690 28026 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd0963c2d344f058b9aa4fd3fc2cce1" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41959 } }
I20260131 03:08:36.076797 28026 leader_election.cc:304] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bd0963c2d344f058b9aa4fd3fc2cce1; no voters:
I20260131 03:08:36.076911 28026 leader_election.cc:290] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:08:36.077034 28029 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:08:36.077169 28026 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:08:36.077196 28029 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1 [term 1 LEADER]: Becoming Leader. State: Replica: 8bd0963c2d344f058b9aa4fd3fc2cce1, State: Running, Role: LEADER
I20260131 03:08:36.077302 28029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd0963c2d344f058b9aa4fd3fc2cce1" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41959 } }
I20260131 03:08:36.077761 28030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bd0963c2d344f058b9aa4fd3fc2cce1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd0963c2d344f058b9aa4fd3fc2cce1" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41959 } } }
I20260131 03:08:36.077862 28030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:36.077771 28031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bd0963c2d344f058b9aa4fd3fc2cce1. Latest consensus state: current_term: 1 leader_uuid: "8bd0963c2d344f058b9aa4fd3fc2cce1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd0963c2d344f058b9aa4fd3fc2cce1" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41959 } } }
I20260131 03:08:36.077901 28031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:36.078190 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 27948
I20260131 03:08:36.078274 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:08:36.078426 19792 external_mini_cluster.cc:1468] Setting key 43c1138e243506fff1c9f1a938621e45
I20260131 03:08:36.079247 28039 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:08:36.079809 28039 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:08:36.082334 28039 catalog_manager.cc:1357] Generated new cluster ID: 95111b55fcad4984a882d3fcc35c0173
I20260131 03:08:36.082410 28039 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Jan 31 03:08:36 dist-test-slave-5nnp krb5kdc[27932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM
I20260131 03:08:36.089561 28040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.079066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56298 (local address 127.19.84.62:41959)
0131 03:08:36.079356 (+ 290us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:36.079369 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:36.079417 (+ 48us) server_negotiation.cc:408] Connection header received
0131 03:08:36.079533 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:36.079539 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:36.079593 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:36.079761 (+ 168us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:36.080730 (+ 969us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.081858 (+ 1128us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.082654 (+ 796us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.082873 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.085638 (+ 2765us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:36.085645 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:36.085645 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:36.085672 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:36.088004 (+ 2332us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:36.088426 (+ 422us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:36.088429 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:36.088430 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:36.088500 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:36.088685 (+ 185us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:36.088686 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:36.088686 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:36.088960 (+ 274us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:36.089120 (+ 160us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:36.089233 (+ 113us) server_negotiation.cc:300] Negotiation successful
0131 03:08:36.089368 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":64,"threads_started":1}
I20260131 03:08:36.102703 28039 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:08:36.103323 28039 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:08:36.111918 28039 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1: Generated new TSK 0
I20260131 03:08:36.112144 28039 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/oryx.keytab.
I20260131 03:08:36.172067 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:41959
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:41293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:36.252776 28054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:36.252974 28054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:36.253013 28054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:36.254424 28054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:36.254494 28054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:36.254529 28054 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260131 03:08:36.254567 28054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:08:36.256253 28054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:41293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.84.62:41959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:36.256515 28054 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:36.256726 28054 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:36.259727 28059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:36.259724 28060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:36.259732 28062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:36.260090 28054 server_base.cc:1060] running on GCE node
I20260131 03:08:36.260528 28054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:36.260792 28054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:36.261961 28054 hybrid_clock.cc:648] HybridClock initialized: now 1769828916261946 us; error 31 us; skew 500 ppm
Jan 31 03:08:36 dist-test-slave-5nnp krb5kdc[27932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828916, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:36.264588 28054 init.cc:377] Logged in from keytab as oryx/127.19.84.1@KRBTEST.COM (short username oryx)
I20260131 03:08:36.265197 28054 webserver.cc:492] Webserver started at http://127.19.84.1:39789/ using document root <none> and password file <none>
I20260131 03:08:36.265362 28054 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:36.265424 28054 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:36.265556 28054 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:36.266537 28054 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "933d0320a7644b8ebac076ae68a53c3f"
format_stamp: "Formatted at 2026-01-31 03:08:36 on dist-test-slave-5nnp"
server_key: "bd0953d20e7907ae8f31878bd7f935ef"
server_key_iv: "158cdcb01f4a35cec46ecc3d5d7034ee"
server_key_version: "encryptionkey@0"
I20260131 03:08:36.266873 28054 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "933d0320a7644b8ebac076ae68a53c3f"
format_stamp: "Formatted at 2026-01-31 03:08:36 on dist-test-slave-5nnp"
server_key: "bd0953d20e7907ae8f31878bd7f935ef"
server_key_iv: "158cdcb01f4a35cec46ecc3d5d7034ee"
server_key_version: "encryptionkey@0"
I20260131 03:08:36.268255 28054 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260131 03:08:36.269023 28069 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:36.269202 28054 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260131 03:08:36.269290 28054 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "933d0320a7644b8ebac076ae68a53c3f"
format_stamp: "Formatted at 2026-01-31 03:08:36 on dist-test-slave-5nnp"
server_key: "bd0953d20e7907ae8f31878bd7f935ef"
server_key_iv: "158cdcb01f4a35cec46ecc3d5d7034ee"
server_key_version: "encryptionkey@0"
I20260131 03:08:36.269374 28054 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:36.285391 28054 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:36.287650 28054 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:36.287815 28054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:36.288475 28054 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:36.288551 28054 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:36.288600 28054 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:36.288630 28054 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 31 03:08:36 dist-test-slave-5nnp krb5kdc[27932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828916, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM
I20260131 03:08:36.296474 28054 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:33971
I20260131 03:08:36.296891 28054 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
I20260131 03:08:36.297418 28184 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:33971 every 8 connection(s)
I20260131 03:08:36.298442 28040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.288888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:45561 (local address 127.19.84.62:41959)
0131 03:08:36.288958 (+ 70us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:36.288961 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:36.289169 (+ 208us) server_negotiation.cc:408] Connection header received
0131 03:08:36.289726 (+ 557us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:36.289728 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:36.289745 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:36.289804 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:36.290895 (+ 1091us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.291684 (+ 789us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.292414 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.292609 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.295118 (+ 2509us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:36.295125 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:36.295126 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:36.295135 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:36.297043 (+ 1908us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:36.297592 (+ 549us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:36.297594 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:36.297594 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:36.297620 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:36.297813 (+ 193us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:36.297814 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:36.297814 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:36.297919 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:36.297976 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:36.298280 (+ 304us) server_negotiation.cc:300] Negotiation successful
0131 03:08:36.298377 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260131 03:08:36.298453 28078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.288881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41959 (local address 127.19.84.1:45561)
0131 03:08:36.289043 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:36.289073 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:36.289617 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:36.289864 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:36.289868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:36.290335 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:36.290799 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.290805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.291806 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.291807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:36.292287 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.292291 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.292531 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.293141 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:36.293169 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:36.295003 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:36.297153 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:36.297160 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:36.297162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:36.297487 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:36.297683 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:36.297684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:36.297684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:36.297749 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:36.297977 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:36.297980 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:36.298110 (+ 130us) client_negotiation.cc:769] Sending connection context
0131 03:08:36.298202 (+ 92us) client_negotiation.cc:240] Negotiation successful
0131 03:08:36.298317 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":99,"threads_started":1}
I20260131 03:08:36.299115 28185 heartbeater.cc:344] Connected to a master server at 127.19.84.62:41959
I20260131 03:08:36.299201 28185 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:36.299458 28185 heartbeater.cc:507] Master 127.19.84.62:41959 requested a full tablet report, sending...
I20260131 03:08:36.300014 27980 ts_manager.cc:194] Registered new tserver with Master: 933d0320a7644b8ebac076ae68a53c3f (127.19.84.1:33971)
I20260131 03:08:36.300835 27980 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.84.1@KRBTEST.COM'} at 127.19.84.1:45561
I20260131 03:08:36.305255 28040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.300320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:56141 (local address 127.19.84.62:41959)
0131 03:08:36.300364 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:36.300366 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:36.300509 (+ 143us) server_negotiation.cc:408] Connection header received
0131 03:08:36.300533 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:36.300534 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:36.300544 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:36.300611 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:36.301012 (+ 401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.301498 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.302033 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.302236 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.302926 (+ 690us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:36.302930 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:36.302931 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:36.302938 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:36.304215 (+ 1277us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:36.304556 (+ 341us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:36.304559 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:36.304560 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:36.304586 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:36.304745 (+ 159us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:36.304746 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:36.304746 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:36.304837 (+ 91us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:36.304911 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:36.305064 (+ 153us) server_negotiation.cc:300] Negotiation successful
0131 03:08:36.305152 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260131 03:08:36.305269 28191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.300266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41959 (local address 127.19.84.1:56141)
0131 03:08:36.300422 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:36.300431 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:36.300488 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:36.300612 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:36.300613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:36.300741 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:36.300915 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.300917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.301570 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.301571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:36.301945 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.301946 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.302035 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.302523 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:36.302527 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:36.302817 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:36.304319 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:36.304320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:36.304320 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:36.304469 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:36.304648 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:36.304649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:36.304649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:36.304682 (+ 33us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:36.304902 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:36.304903 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:36.304951 (+ 48us) client_negotiation.cc:769] Sending connection context
0131 03:08:36.305040 (+ 89us) client_negotiation.cc:240] Negotiation successful
0131 03:08:36.305152 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":71,"threads_started":1}
I20260131 03:08:36.306934 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 28054
I20260131 03:08:36.307016 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:08:36.307137 19792 external_mini_cluster.cc:1468] Setting key 972379f824532d84a51bada1fdd31fc5
WARNING: no policy specified for oryx/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/oryx.keytab.
I20260131 03:08:36.359984 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:41959
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:41293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:36.440671 28196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:36.440866 28196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:36.440892 28196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:36.442420 28196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:36.442487 28196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:36.442512 28196 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260131 03:08:36.442530 28196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:08:36.444366 28196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:41293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.84.62:41959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:36.444650 28196 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:36.444857 28196 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260131 03:08:36.447839 28196 server_base.cc:1060] running on GCE node
W20260131 03:08:36.447817 28204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:36.447849 28202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:36.448103 28201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:36.448309 28196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:36.448565 28196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:36.449723 28196 hybrid_clock.cc:648] HybridClock initialized: now 1769828916449726 us; error 46 us; skew 500 ppm
Jan 31 03:08:36 dist-test-slave-5nnp krb5kdc[27932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828916, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:36.452360 28196 init.cc:377] Logged in from keytab as oryx/127.19.84.2@KRBTEST.COM (short username oryx)
I20260131 03:08:36.452881 28196 webserver.cc:492] Webserver started at http://127.19.84.2:41693/ using document root <none> and password file <none>
I20260131 03:08:36.453102 28196 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:36.453141 28196 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:36.453258 28196 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:36.454245 28196 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "d0fb784728cb414594ee1ee6d03e875b"
format_stamp: "Formatted at 2026-01-31 03:08:36 on dist-test-slave-5nnp"
server_key: "c52ee37dd3ead1d30d289119650f9521"
server_key_iv: "bd83ba287886aaedd99766de2e1828d9"
server_key_version: "encryptionkey@0"
I20260131 03:08:36.454577 28196 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "d0fb784728cb414594ee1ee6d03e875b"
format_stamp: "Formatted at 2026-01-31 03:08:36 on dist-test-slave-5nnp"
server_key: "c52ee37dd3ead1d30d289119650f9521"
server_key_iv: "bd83ba287886aaedd99766de2e1828d9"
server_key_version: "encryptionkey@0"
I20260131 03:08:36.455935 28196 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:08:36.456841 28211 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:36.457094 28196 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:36.457190 28196 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "d0fb784728cb414594ee1ee6d03e875b"
format_stamp: "Formatted at 2026-01-31 03:08:36 on dist-test-slave-5nnp"
server_key: "c52ee37dd3ead1d30d289119650f9521"
server_key_iv: "bd83ba287886aaedd99766de2e1828d9"
server_key_version: "encryptionkey@0"
I20260131 03:08:36.457275 28196 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:36.474589 28196 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:36.476665 28196 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:36.476840 28196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:36.477380 28196 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:36.477443 28196 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:36.477527 28196 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:36.477559 28196 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 31 03:08:36 dist-test-slave-5nnp krb5kdc[27932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828916, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM
I20260131 03:08:36.484740 28196 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:38709
I20260131 03:08:36.484844 28326 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:38709 every 8 connection(s)
I20260131 03:08:36.485097 28196 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
I20260131 03:08:36.488662 28040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.477946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:56265 (local address 127.19.84.62:41959)
0131 03:08:36.478017 (+ 71us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:36.478019 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:36.478230 (+ 211us) server_negotiation.cc:408] Connection header received
0131 03:08:36.479095 (+ 865us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:36.479097 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:36.479110 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:36.479173 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:36.480358 (+ 1185us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.481169 (+ 811us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.482198 (+ 1029us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.482399 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.485369 (+ 2970us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:36.485375 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:36.485376 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:36.485385 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:36.486911 (+ 1526us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:36.487346 (+ 435us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:36.487348 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:36.487349 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:36.487419 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:36.487821 (+ 402us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:36.487823 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:36.487823 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:36.487944 (+ 121us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:36.488024 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:36.488428 (+ 404us) server_negotiation.cc:300] Negotiation successful
0131 03:08:36.488558 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260131 03:08:36.488754 28220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.477877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41959 (local address 127.19.84.2:56265)
0131 03:08:36.478072 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:36.478135 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:36.478924 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:36.479248 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:36.479260 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:36.479721 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:36.480204 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.480209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.481263 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.481264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:36.481813 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.481816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.481965 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.482505 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:36.482535 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:36.485254 (+ 2719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:36.487024 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:36.487031 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:36.487034 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:36.487254 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:36.487513 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:36.487514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:36.487515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:36.487679 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:36.488015 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:36.488021 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:36.488250 (+ 229us) client_negotiation.cc:769] Sending connection context
0131 03:08:36.488406 (+ 156us) client_negotiation.cc:240] Negotiation successful
0131 03:08:36.488562 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":79,"threads_started":1}
I20260131 03:08:36.489454 28327 heartbeater.cc:344] Connected to a master server at 127.19.84.62:41959
I20260131 03:08:36.489544 28327 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:36.489794 28327 heartbeater.cc:507] Master 127.19.84.62:41959 requested a full tablet report, sending...
I20260131 03:08:36.490260 27979 ts_manager.cc:194] Registered new tserver with Master: d0fb784728cb414594ee1ee6d03e875b (127.19.84.2:38709)
I20260131 03:08:36.490777 27979 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.84.2@KRBTEST.COM'} at 127.19.84.2:56265
I20260131 03:08:36.494911 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 28196
I20260131 03:08:36.495038 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:08:36.495223 19792 external_mini_cluster.cc:1468] Setting key ef04c957f9c0fbf92702bb334f25bf0b
I20260131 03:08:36.496044 28040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.490689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:35083 (local address 127.19.84.62:41959)
0131 03:08:36.490730 (+ 41us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:36.490732 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:36.490873 (+ 141us) server_negotiation.cc:408] Connection header received
0131 03:08:36.490933 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:36.490933 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:36.490944 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:36.490996 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:36.491491 (+ 495us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.491955 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.492505 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.492696 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.493404 (+ 708us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:36.493408 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:36.493408 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:36.493416 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:36.494780 (+ 1364us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:36.495224 (+ 444us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:36.495225 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:36.495225 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:36.495253 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:36.495506 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:36.495507 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:36.495508 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:36.495637 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:36.495718 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:36.495863 (+ 145us) server_negotiation.cc:300] Negotiation successful
0131 03:08:36.495950 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260131 03:08:36.496060 28333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.490630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41959 (local address 127.19.84.2:35083)
0131 03:08:36.490791 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:36.490802 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:36.490868 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:36.491013 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:36.491013 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:36.491154 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:36.491334 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.491336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.492041 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.492042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:36.492419 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.492420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.492509 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.493040 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:36.493049 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:36.493298 (+ 249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:36.494876 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:36.494877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:36.494877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:36.495023 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:36.495342 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:36.495343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:36.495344 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:36.495424 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:36.495708 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:36.495709 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:36.495763 (+ 54us) client_negotiation.cc:769] Sending connection context
0131 03:08:36.495840 (+ 77us) client_negotiation.cc:240] Negotiation successful
0131 03:08:36.495950 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for oryx/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/oryx.keytab.
I20260131 03:08:36.548141 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:41959
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:41293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:36.629107 28338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:36.629307 28338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:36.629345 28338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:36.630766 28338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:36.630836 28338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:36.630867 28338 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260131 03:08:36.630909 28338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:08:36.632606 28338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:41293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/krb5kdc/oryx.keytab
--principal=oryx/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.84.62:41959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:36.632854 28338 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:36.633041 28338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:36.635581 28343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:36.635566 28338 server_base.cc:1060] running on GCE node
W20260131 03:08:36.635555 28346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:36.635572 28344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:36.636001 28338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:36.636214 28338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:36.637346 28338 hybrid_clock.cc:648] HybridClock initialized: now 1769828916637342 us; error 38 us; skew 500 ppm
Jan 31 03:08:36 dist-test-slave-5nnp krb5kdc[27932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828916, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:36.639879 28338 init.cc:377] Logged in from keytab as oryx/127.19.84.3@KRBTEST.COM (short username oryx)
I20260131 03:08:36.640430 28338 webserver.cc:492] Webserver started at http://127.19.84.3:36215/ using document root <none> and password file <none>
I20260131 03:08:36.640626 28338 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:36.640662 28338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:36.640792 28338 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:36.641697 28338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "6cea691cf09a44fda4b6307ef400cbd6"
format_stamp: "Formatted at 2026-01-31 03:08:36 on dist-test-slave-5nnp"
server_key: "0af466a3d873584990f124bec4396374"
server_key_iv: "8722e3841a9936ab7ce2059fc0f00100"
server_key_version: "encryptionkey@0"
I20260131 03:08:36.642062 28338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "6cea691cf09a44fda4b6307ef400cbd6"
format_stamp: "Formatted at 2026-01-31 03:08:36 on dist-test-slave-5nnp"
server_key: "0af466a3d873584990f124bec4396374"
server_key_iv: "8722e3841a9936ab7ce2059fc0f00100"
server_key_version: "encryptionkey@0"
I20260131 03:08:36.643244 28338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260131 03:08:36.643927 28353 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:36.644111 28338 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:36.644196 28338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "6cea691cf09a44fda4b6307ef400cbd6"
format_stamp: "Formatted at 2026-01-31 03:08:36 on dist-test-slave-5nnp"
server_key: "0af466a3d873584990f124bec4396374"
server_key_iv: "8722e3841a9936ab7ce2059fc0f00100"
server_key_version: "encryptionkey@0"
I20260131 03:08:36.644273 28338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:36.661707 28338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:36.663744 28338 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:36.663899 28338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:36.664474 28338 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:36.664530 28338 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:36.664572 28338 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:36.664592 28338 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 31 03:08:36 dist-test-slave-5nnp krb5kdc[27932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828916, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM
I20260131 03:08:36.672058 28338 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:36069
I20260131 03:08:36.672132 28468 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:36069 every 8 connection(s)
I20260131 03:08:36.672477 28338 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
I20260131 03:08:36.672650 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 28338
I20260131 03:08:36.672729 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:08:36.672920 19792 external_mini_cluster.cc:1468] Setting key 20de4c89f2597263badb0e94ee13495e
I20260131 03:08:36.673913 28040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.664861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:40189 (local address 127.19.84.62:41959)
0131 03:08:36.664932 (+ 71us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:36.664935 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:36.665163 (+ 228us) server_negotiation.cc:408] Connection header received
0131 03:08:36.665940 (+ 777us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:36.665941 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:36.665955 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:36.666017 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:36.667146 (+ 1129us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.667844 (+ 698us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.668491 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.668676 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.670671 (+ 1995us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:36.670678 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:36.670679 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:36.670691 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:36.672301 (+ 1610us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:36.672749 (+ 448us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:36.672750 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:36.672751 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:36.672777 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:36.673020 (+ 243us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:36.673021 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:36.673022 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:36.673163 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:36.673383 (+ 220us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:36.673714 (+ 331us) server_negotiation.cc:300] Negotiation successful
0131 03:08:36.673815 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260131 03:08:36.674011 28362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.664844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41959 (local address 127.19.84.3:40189)
0131 03:08:36.665030 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:36.665067 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:36.665808 (+ 741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:36.666083 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:36.666087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:36.666564 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:36.667012 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.667016 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.667954 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.667956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:36.668366 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.668368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.668560 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.669120 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:36.669138 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:36.670567 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:36.672438 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:36.672447 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:36.672449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:36.672638 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:36.672854 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:36.672855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:36.672855 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:36.672959 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:36.673370 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:36.673375 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:36.673548 (+ 173us) client_negotiation.cc:769] Sending connection context
0131 03:08:36.673695 (+ 147us) client_negotiation.cc:240] Negotiation successful
0131 03:08:36.673834 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":101,"threads_started":1}
I20260131 03:08:36.674429 28469 heartbeater.cc:344] Connected to a master server at 127.19.84.62:41959
I20260131 03:08:36.674535 28469 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:36.674763 28469 heartbeater.cc:507] Master 127.19.84.62:41959 requested a full tablet report, sending...
I20260131 03:08:36.675175 27980 ts_manager.cc:194] Registered new tserver with Master: 6cea691cf09a44fda4b6307ef400cbd6 (127.19.84.3:36069)
I20260131 03:08:36.675688 27980 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:40189
I20260131 03:08:36.680428 28040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.675597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:51819 (local address 127.19.84.62:41959)
0131 03:08:36.675634 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:36.675636 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:36.675935 (+ 299us) server_negotiation.cc:408] Connection header received
0131 03:08:36.675964 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:36.675965 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:36.675976 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:36.676038 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:36.676564 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.677030 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.677527 (+ 497us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.677705 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.678350 (+ 645us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:36.678353 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:36.678353 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:36.678358 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:36.679678 (+ 1320us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:36.679958 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:36.679958 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:36.679959 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:36.679977 (+ 18us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:36.680089 (+ 112us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:36.680089 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:36.680090 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:36.680160 (+ 70us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:36.680203 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:36.680307 (+ 104us) server_negotiation.cc:300] Negotiation successful
0131 03:08:36.680364 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260131 03:08:36.680449 28475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.675525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41959 (local address 127.19.84.3:51819)
0131 03:08:36.675857 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:36.675868 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:36.675929 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:36.676050 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:36.676051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:36.676213 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:36.676460 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.676462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.677103 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.677104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:36.677457 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.677458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.677524 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.678034 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:36.678038 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:36.678281 (+ 243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:36.679756 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:36.679756 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:36.679757 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:36.679898 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:36.680030 (+ 132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:36.680030 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:36.680031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:36.680054 (+ 23us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:36.680210 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:36.680211 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:36.680244 (+ 33us) client_negotiation.cc:769] Sending connection context
0131 03:08:36.680294 (+ 50us) client_negotiation.cc:240] Negotiation successful
0131 03:08:36.680371 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":64,"threads_started":1}
I20260131 03:08:36.684908 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Jan 31 03:08:36 dist-test-slave-5nnp krb5kdc[27932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828916, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 31 03:08:36 dist-test-slave-5nnp krb5kdc[27932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:36 dist-test-slave-5nnp krb5kdc[27932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
W20260131 03:08:36.705174 28040 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:36.700101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56312 (local address 127.19.84.62:41959)
0131 03:08:36.700165 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:36.700167 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:36.700183 (+ 16us) server_negotiation.cc:408] Connection header received
0131 03:08:36.700210 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:36.700211 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:36.700221 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:36.700332 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:36.701066 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.701559 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.702111 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.702227 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.705077 (+ 2850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56312: BlockingRecv error: recv got EOF from 127.0.0.1:56312 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jan 31 03:08:36 dist-test-slave-5nnp krb5kdc[27932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828916, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM
I20260131 03:08:36.712868 28040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.706220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56322 (local address 127.19.84.62:41959)
0131 03:08:36.706297 (+ 77us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:36.706299 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:36.706386 (+ 87us) server_negotiation.cc:408] Connection header received
0131 03:08:36.706532 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:36.706534 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:36.706544 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:36.706615 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:36.707219 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.707768 (+ 549us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.708317 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.708508 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.710520 (+ 2012us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:36.710523 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:36.710524 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:36.710530 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:36.711777 (+ 1247us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:36.712250 (+ 473us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:36.712251 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:36.712251 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:36.712270 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:36.712425 (+ 155us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:36.712425 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:36.712426 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:36.712527 (+ 101us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:36.712606 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:36.712737 (+ 131us) server_negotiation.cc:300] Negotiation successful
0131 03:08:36.712799 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260131 03:08:36.714524 27980 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:08:36.715896 27980 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:08:36.721355 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.718150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:38709 (local address 127.0.0.1:43784)
0131 03:08:36.718349 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:36.718358 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:36.718438 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:36.719072 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:36.719073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:36.719074 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:36.719260 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.719261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.720249 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.720250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:36.721091 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.721093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.721163 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.721199 (+ 36us) client_negotiation.cc:769] Sending connection context
0131 03:08:36.721247 (+ 48us) client_negotiation.cc:240] Negotiation successful
0131 03:08:36.721281 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":95,"threads_started":1}
I20260131 03:08:36.721611 28495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.718358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:33971 (local address 127.0.0.1:45880)
0131 03:08:36.718487 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:36.718500 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:36.718561 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:36.718983 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:36.718984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:36.718987 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:36.719225 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.719226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.720566 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.720567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:36.721410 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.721413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.721476 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.721479 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:08:36.721510 (+ 31us) client_negotiation.cc:240] Negotiation successful
0131 03:08:36.721549 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":87,"threads_started":1}
I20260131 03:08:36.721927 28491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.718121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:36069 (local address 127.0.0.1:43906)
0131 03:08:36.718319 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:36.718352 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:36.718438 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:36.718782 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:36.718785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:36.718788 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:36.719196 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.719198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.720583 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.720584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:36.721327 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.721329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.721649 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.721771 (+ 122us) client_negotiation.cc:769] Sending connection context
0131 03:08:36.721820 (+ 49us) client_negotiation.cc:240] Negotiation successful
0131 03:08:36.721860 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":79,"threads_started":1}
I20260131 03:08:36.722208 28493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.718208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43906 (local address 127.19.84.3:36069)
0131 03:08:36.718593 (+ 385us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:36.718601 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:36.718612 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:08:36.718651 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:36.718654 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:36.718673 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:36.718754 (+ 81us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:36.719295 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.720479 (+ 1184us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.721445 (+ 966us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.721972 (+ 527us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.722036 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:36.722067 (+ 31us) server_negotiation.cc:300] Negotiation successful
0131 03:08:36.722109 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":86,"threads_started":1}
I20260131 03:08:36.722330 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.718254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43784 (local address 127.19.84.2:38709)
0131 03:08:36.718895 (+ 641us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:36.718901 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:36.718909 (+ 8us) server_negotiation.cc:408] Connection header received
0131 03:08:36.718960 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:36.718962 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:36.718977 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:36.719339 (+ 362us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:36.719477 (+ 138us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.720159 (+ 682us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.721593 (+ 1434us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.722136 (+ 543us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.722182 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:36.722213 (+ 31us) server_negotiation.cc:300] Negotiation successful
0131 03:08:36.722253 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":76,"threads_started":1}
I20260131 03:08:36.722788 28403 tablet_service.cc:1505] Processing CreateTablet for tablet f442ed3ece1543318da6d79fd2cf431d (DEFAULT_TABLE table=test-table [id=19fd5cd8ccdd439d8d4dbe1c908b5ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:08:36.722820 28261 tablet_service.cc:1505] Processing CreateTablet for tablet f442ed3ece1543318da6d79fd2cf431d (DEFAULT_TABLE table=test-table [id=19fd5cd8ccdd439d8d4dbe1c908b5ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:08:36.723093 28261 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f442ed3ece1543318da6d79fd2cf431d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:36.723096 28403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f442ed3ece1543318da6d79fd2cf431d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:36.723673 28496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.718613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45880 (local address 127.19.84.1:33971)
0131 03:08:36.718818 (+ 205us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:36.718825 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:36.718835 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:08:36.718870 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:36.718873 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:36.718888 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:36.718962 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:36.719298 (+ 336us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.720483 (+ 1185us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.721798 (+ 1315us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.722288 (+ 490us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.723497 (+ 1209us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:36.723530 (+ 33us) server_negotiation.cc:300] Negotiation successful
0131 03:08:36.723594 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":94,"threads_started":1}
I20260131 03:08:36.725533 28498 tablet_bootstrap.cc:492] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6: Bootstrap starting.
I20260131 03:08:36.725550 28497 tablet_bootstrap.cc:492] T f442ed3ece1543318da6d79fd2cf431d P d0fb784728cb414594ee1ee6d03e875b: Bootstrap starting.
I20260131 03:08:36.725970 28119 tablet_service.cc:1505] Processing CreateTablet for tablet f442ed3ece1543318da6d79fd2cf431d (DEFAULT_TABLE table=test-table [id=19fd5cd8ccdd439d8d4dbe1c908b5ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:08:36.726192 28498 tablet_bootstrap.cc:654] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:36.726271 28119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f442ed3ece1543318da6d79fd2cf431d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:36.726305 28497 tablet_bootstrap.cc:654] T f442ed3ece1543318da6d79fd2cf431d P d0fb784728cb414594ee1ee6d03e875b: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:36.726795 28497 log.cc:826] T f442ed3ece1543318da6d79fd2cf431d P d0fb784728cb414594ee1ee6d03e875b: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:36.726878 28498 log.cc:826] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:36.727648 28497 tablet_bootstrap.cc:492] T f442ed3ece1543318da6d79fd2cf431d P d0fb784728cb414594ee1ee6d03e875b: No bootstrap required, opened a new log
I20260131 03:08:36.727648 28498 tablet_bootstrap.cc:492] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6: No bootstrap required, opened a new log
I20260131 03:08:36.727721 28497 ts_tablet_manager.cc:1403] T f442ed3ece1543318da6d79fd2cf431d P d0fb784728cb414594ee1ee6d03e875b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:08:36.727732 28498 ts_tablet_manager.cc:1403] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:08:36.728937 28501 tablet_bootstrap.cc:492] T f442ed3ece1543318da6d79fd2cf431d P 933d0320a7644b8ebac076ae68a53c3f: Bootstrap starting.
I20260131 03:08:36.729344 28498 raft_consensus.cc:359] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } }
I20260131 03:08:36.729506 28498 raft_consensus.cc:385] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:36.729434 28497 raft_consensus.cc:359] T f442ed3ece1543318da6d79fd2cf431d P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } }
I20260131 03:08:36.729539 28498 raft_consensus.cc:740] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cea691cf09a44fda4b6307ef400cbd6, State: Initialized, Role: FOLLOWER
I20260131 03:08:36.729562 28497 raft_consensus.cc:385] T f442ed3ece1543318da6d79fd2cf431d P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:36.729589 28497 raft_consensus.cc:740] T f442ed3ece1543318da6d79fd2cf431d P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0fb784728cb414594ee1ee6d03e875b, State: Initialized, Role: FOLLOWER
I20260131 03:08:36.729687 28497 consensus_queue.cc:260] T f442ed3ece1543318da6d79fd2cf431d P d0fb784728cb414594ee1ee6d03e875b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } }
I20260131 03:08:36.729825 28498 consensus_queue.cc:260] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } }
I20260131 03:08:36.729676 28501 tablet_bootstrap.cc:654] T f442ed3ece1543318da6d79fd2cf431d P 933d0320a7644b8ebac076ae68a53c3f: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:36.730049 28498 ts_tablet_manager.cc:1434] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260131 03:08:36.730171 28469 heartbeater.cc:499] Master 127.19.84.62:41959 was elected leader, sending a full tablet report...
I20260131 03:08:36.730885 28501 log.cc:826] T f442ed3ece1543318da6d79fd2cf431d P 933d0320a7644b8ebac076ae68a53c3f: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:36.731706 28497 ts_tablet_manager.cc:1434] T f442ed3ece1543318da6d79fd2cf431d P d0fb784728cb414594ee1ee6d03e875b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260131 03:08:36.731849 28327 heartbeater.cc:499] Master 127.19.84.62:41959 was elected leader, sending a full tablet report...
I20260131 03:08:36.733917 28501 tablet_bootstrap.cc:492] T f442ed3ece1543318da6d79fd2cf431d P 933d0320a7644b8ebac076ae68a53c3f: No bootstrap required, opened a new log
I20260131 03:08:36.734007 28501 ts_tablet_manager.cc:1403] T f442ed3ece1543318da6d79fd2cf431d P 933d0320a7644b8ebac076ae68a53c3f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260131 03:08:36.735153 28501 raft_consensus.cc:359] T f442ed3ece1543318da6d79fd2cf431d P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } }
I20260131 03:08:36.735307 28501 raft_consensus.cc:385] T f442ed3ece1543318da6d79fd2cf431d P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:36.735349 28501 raft_consensus.cc:740] T f442ed3ece1543318da6d79fd2cf431d P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 933d0320a7644b8ebac076ae68a53c3f, State: Initialized, Role: FOLLOWER
I20260131 03:08:36.735483 28501 consensus_queue.cc:260] T f442ed3ece1543318da6d79fd2cf431d P 933d0320a7644b8ebac076ae68a53c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } }
W20260131 03:08:36.735745 28328 tablet.cc:2378] T f442ed3ece1543318da6d79fd2cf431d P d0fb784728cb414594ee1ee6d03e875b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:08:36.735783 28501 ts_tablet_manager.cc:1434] T f442ed3ece1543318da6d79fd2cf431d P 933d0320a7644b8ebac076ae68a53c3f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:08:36.736002 28185 heartbeater.cc:499] Master 127.19.84.62:41959 was elected leader, sending a full tablet report...
W20260131 03:08:36.797737 28186 tablet.cc:2378] T f442ed3ece1543318da6d79fd2cf431d P 933d0320a7644b8ebac076ae68a53c3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:08:36.862010 28503 raft_consensus.cc:493] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:08:36.862139 28503 raft_consensus.cc:515] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } }
I20260131 03:08:36.862464 28503 leader_election.cc:290] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0fb784728cb414594ee1ee6d03e875b (127.19.84.2:38709), 933d0320a7644b8ebac076ae68a53c3f (127.19.84.1:33971)
I20260131 03:08:36.864877 28362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.862712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:33971 (local address 127.19.84.3:57785)
0131 03:08:36.862780 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:36.862793 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:36.862842 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:36.863038 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:36.863039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:36.863042 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:36.863262 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.863264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.864046 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.864047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:36.864701 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.864703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.864765 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.864769 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:08:36.864792 (+ 23us) client_negotiation.cc:240] Negotiation successful
0131 03:08:36.864819 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260131 03:08:36.865444 28496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.862787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:57785 (local address 127.19.84.1:33971)
0131 03:08:36.862905 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:36.862908 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:36.862918 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:08:36.862950 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:36.862952 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:36.862963 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:36.863021 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:36.863370 (+ 349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.863964 (+ 594us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.864819 (+ 855us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.865307 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.865317 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:36.865340 (+ 23us) server_negotiation.cc:300] Negotiation successful
0131 03:08:36.865374 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260131 03:08:36.865763 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.862810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:42741 (local address 127.19.84.2:38709)
0131 03:08:36.862855 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:36.862857 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:36.862987 (+ 130us) server_negotiation.cc:408] Connection header received
0131 03:08:36.863039 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:36.863040 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:36.863052 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:36.863109 (+ 57us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:36.863581 (+ 472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.864124 (+ 543us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.865027 (+ 903us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.865496 (+ 469us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.865505 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:36.865641 (+ 136us) server_negotiation.cc:300] Negotiation successful
0131 03:08:36.865687 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260131 03:08:36.865777 28506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.862736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:38709 (local address 127.19.84.3:42741)
0131 03:08:36.862873 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:36.862885 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:36.862960 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:36.863150 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:36.863151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:36.863154 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:36.863487 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.863491 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.864223 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.864224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:36.864913 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.864915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.865537 (+ 622us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.865541 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:08:36.865633 (+ 92us) client_negotiation.cc:240] Negotiation successful
0131 03:08:36.865678 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260131 03:08:36.865844 28139 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f442ed3ece1543318da6d79fd2cf431d" candidate_uuid: "6cea691cf09a44fda4b6307ef400cbd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "933d0320a7644b8ebac076ae68a53c3f" is_pre_election: true
I20260131 03:08:36.866047 28139 raft_consensus.cc:2468] T f442ed3ece1543318da6d79fd2cf431d P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cea691cf09a44fda4b6307ef400cbd6 in term 0.
I20260131 03:08:36.866115 28281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f442ed3ece1543318da6d79fd2cf431d" candidate_uuid: "6cea691cf09a44fda4b6307ef400cbd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0fb784728cb414594ee1ee6d03e875b" is_pre_election: true
I20260131 03:08:36.866277 28281 raft_consensus.cc:2468] T f442ed3ece1543318da6d79fd2cf431d P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cea691cf09a44fda4b6307ef400cbd6 in term 0.
I20260131 03:08:36.866317 28356 leader_election.cc:304] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cea691cf09a44fda4b6307ef400cbd6, 933d0320a7644b8ebac076ae68a53c3f; no voters:
I20260131 03:08:36.866506 28503 raft_consensus.cc:2804] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:08:36.866593 28503 raft_consensus.cc:493] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:08:36.866632 28503 raft_consensus.cc:3060] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:36.867308 28503 raft_consensus.cc:515] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } }
I20260131 03:08:36.867542 28503 leader_election.cc:290] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6 [CANDIDATE]: Term 1 election: Requested vote from peers d0fb784728cb414594ee1ee6d03e875b (127.19.84.2:38709), 933d0320a7644b8ebac076ae68a53c3f (127.19.84.1:33971)
I20260131 03:08:36.867753 28281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f442ed3ece1543318da6d79fd2cf431d" candidate_uuid: "6cea691cf09a44fda4b6307ef400cbd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0fb784728cb414594ee1ee6d03e875b"
I20260131 03:08:36.867745 28139 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f442ed3ece1543318da6d79fd2cf431d" candidate_uuid: "6cea691cf09a44fda4b6307ef400cbd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "933d0320a7644b8ebac076ae68a53c3f"
I20260131 03:08:36.867831 28281 raft_consensus.cc:3060] T f442ed3ece1543318da6d79fd2cf431d P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:36.867831 28139 raft_consensus.cc:3060] T f442ed3ece1543318da6d79fd2cf431d P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:36.868755 28281 raft_consensus.cc:2468] T f442ed3ece1543318da6d79fd2cf431d P d0fb784728cb414594ee1ee6d03e875b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cea691cf09a44fda4b6307ef400cbd6 in term 1.
I20260131 03:08:36.868755 28139 raft_consensus.cc:2468] T f442ed3ece1543318da6d79fd2cf431d P 933d0320a7644b8ebac076ae68a53c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cea691cf09a44fda4b6307ef400cbd6 in term 1.
I20260131 03:08:36.868928 28356 leader_election.cc:304] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cea691cf09a44fda4b6307ef400cbd6, 933d0320a7644b8ebac076ae68a53c3f; no voters:
I20260131 03:08:36.869020 28503 raft_consensus.cc:2804] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:08:36.869153 28503 raft_consensus.cc:697] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6 [term 1 LEADER]: Becoming Leader. State: Replica: 6cea691cf09a44fda4b6307ef400cbd6, State: Running, Role: LEADER
I20260131 03:08:36.869256 28503 consensus_queue.cc:237] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } }
I20260131 03:08:36.870277 27980 catalog_manager.cc:5654] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cea691cf09a44fda4b6307ef400cbd6 (127.19.84.3). New cstate: current_term: 1 leader_uuid: "6cea691cf09a44fda4b6307ef400cbd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } health_report { overall_health: UNKNOWN } } }
I20260131 03:08:36.910218 28493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.907550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43918 (local address 127.19.84.3:36069)
0131 03:08:36.907619 (+ 69us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:36.907622 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:36.907632 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:08:36.907692 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:36.907693 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:36.907719 (+ 26us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:36.907806 (+ 87us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:08:36.908360 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.908879 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.909548 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.909717 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.909755 (+ 38us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:08:36.909853 (+ 98us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:08:36.909921 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:36.910041 (+ 120us) server_negotiation.cc:300] Negotiation successful
0131 03:08:36.910069 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260131 03:08:36.912461 28139 raft_consensus.cc:1275] T f442ed3ece1543318da6d79fd2cf431d P 933d0320a7644b8ebac076ae68a53c3f [term 1 FOLLOWER]: Refusing update from remote peer 6cea691cf09a44fda4b6307ef400cbd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:08:36.912712 28281 raft_consensus.cc:1275] T f442ed3ece1543318da6d79fd2cf431d P d0fb784728cb414594ee1ee6d03e875b [term 1 FOLLOWER]: Refusing update from remote peer 6cea691cf09a44fda4b6307ef400cbd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:08:36.912876 28507 consensus_queue.cc:1048] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:08:36.913081 28503 consensus_queue.cc:1048] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:08:36.916298 28511 mvcc.cc:204] Tried to move back new op lower bound from 7249219243670446080 to 7249219243497234432. Current Snapshot: MvccSnapshot[applied={T|T < 7249219243670446080}]
I20260131 03:08:36.919126 27979 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:08:36.919250 27979 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:08:36.920333 27979 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1: Sending DeleteTablet for 3 replicas of tablet f442ed3ece1543318da6d79fd2cf431d
I20260131 03:08:36.920759 28119 tablet_service.cc:1552] Processing DeleteTablet for tablet f442ed3ece1543318da6d79fd2cf431d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:36 UTC) from {username='oryx', principal='oryx/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:45880
I20260131 03:08:36.920753 28403 tablet_service.cc:1552] Processing DeleteTablet for tablet f442ed3ece1543318da6d79fd2cf431d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:36 UTC) from {username='oryx', principal='oryx/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:43906
I20260131 03:08:36.921057 28261 tablet_service.cc:1552] Processing DeleteTablet for tablet f442ed3ece1543318da6d79fd2cf431d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:36 UTC) from {username='oryx', principal='oryx/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:43784
I20260131 03:08:36.921221 28520 tablet_replica.cc:333] T f442ed3ece1543318da6d79fd2cf431d P 933d0320a7644b8ebac076ae68a53c3f: stopping tablet replica
I20260131 03:08:36.921277 28522 tablet_replica.cc:333] T f442ed3ece1543318da6d79fd2cf431d P d0fb784728cb414594ee1ee6d03e875b: stopping tablet replica
I20260131 03:08:36.921341 28522 raft_consensus.cc:2243] T f442ed3ece1543318da6d79fd2cf431d P d0fb784728cb414594ee1ee6d03e875b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:08:36.921336 27979 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260131 03:08:36.921465 28522 raft_consensus.cc:2272] T f442ed3ece1543318da6d79fd2cf431d P d0fb784728cb414594ee1ee6d03e875b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260131 03:08:36.921509 27979 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:08:36.921826 28520 raft_consensus.cc:2243] T f442ed3ece1543318da6d79fd2cf431d P 933d0320a7644b8ebac076ae68a53c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:08:36.921957 28520 raft_consensus.cc:2272] T f442ed3ece1543318da6d79fd2cf431d P 933d0320a7644b8ebac076ae68a53c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:08:36.922151 28521 tablet_replica.cc:333] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6: stopping tablet replica
I20260131 03:08:36.922236 28521 raft_consensus.cc:2243] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:08:36.922367 28521 raft_consensus.cc:2272] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:08:36.923014 28521 ts_tablet_manager.cc:1916] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:08:36.923110 28119 tablet_service.cc:1505] Processing CreateTablet for tablet 98aa6d511d634562b7ff45c12f617bcd (DEFAULT_TABLE table=test-table [id=3f4f68e88485428abd2341b997041a6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:08:36.923230 28119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98aa6d511d634562b7ff45c12f617bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:36.923251 28520 ts_tablet_manager.cc:1916] T f442ed3ece1543318da6d79fd2cf431d P 933d0320a7644b8ebac076ae68a53c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:08:36.924710 28501 tablet_bootstrap.cc:492] T 98aa6d511d634562b7ff45c12f617bcd P 933d0320a7644b8ebac076ae68a53c3f: Bootstrap starting.
I20260131 03:08:36.924769 28403 tablet_service.cc:1505] Processing CreateTablet for tablet 98aa6d511d634562b7ff45c12f617bcd (DEFAULT_TABLE table=test-table [id=3f4f68e88485428abd2341b997041a6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:08:36.924906 28403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98aa6d511d634562b7ff45c12f617bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:36.925176 28501 tablet_bootstrap.cc:654] T 98aa6d511d634562b7ff45c12f617bcd P 933d0320a7644b8ebac076ae68a53c3f: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:36.925731 28521 ts_tablet_manager.cc:1929] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:08:36.925815 28521 log.cc:1199] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/f442ed3ece1543318da6d79fd2cf431d
I20260131 03:08:36.926059 28520 ts_tablet_manager.cc:1929] T f442ed3ece1543318da6d79fd2cf431d P 933d0320a7644b8ebac076ae68a53c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:08:36.926075 28521 ts_tablet_manager.cc:1950] T f442ed3ece1543318da6d79fd2cf431d P 6cea691cf09a44fda4b6307ef400cbd6: Deleting consensus metadata
I20260131 03:08:36.926445 28498 tablet_bootstrap.cc:492] T 98aa6d511d634562b7ff45c12f617bcd P 6cea691cf09a44fda4b6307ef400cbd6: Bootstrap starting.
I20260131 03:08:36.926517 28261 tablet_service.cc:1505] Processing CreateTablet for tablet 98aa6d511d634562b7ff45c12f617bcd (DEFAULT_TABLE table=test-table [id=3f4f68e88485428abd2341b997041a6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:08:36.926694 28261 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98aa6d511d634562b7ff45c12f617bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:36.926898 28498 tablet_bootstrap.cc:654] T 98aa6d511d634562b7ff45c12f617bcd P 6cea691cf09a44fda4b6307ef400cbd6: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:36.926990 27966 catalog_manager.cc:4985] TS 6cea691cf09a44fda4b6307ef400cbd6 (127.19.84.3:36069): tablet f442ed3ece1543318da6d79fd2cf431d (table test-table [id=19fd5cd8ccdd439d8d4dbe1c908b5ec3]) successfully deleted
I20260131 03:08:36.927556 28520 log.cc:1199] T f442ed3ece1543318da6d79fd2cf431d P 933d0320a7644b8ebac076ae68a53c3f: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-0/wal/wals/f442ed3ece1543318da6d79fd2cf431d
I20260131 03:08:36.927592 28498 tablet_bootstrap.cc:492] T 98aa6d511d634562b7ff45c12f617bcd P 6cea691cf09a44fda4b6307ef400cbd6: No bootstrap required, opened a new log
I20260131 03:08:36.927644 28498 ts_tablet_manager.cc:1403] T 98aa6d511d634562b7ff45c12f617bcd P 6cea691cf09a44fda4b6307ef400cbd6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260131 03:08:36.927816 28498 raft_consensus.cc:359] T 98aa6d511d634562b7ff45c12f617bcd P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:36.927906 28498 raft_consensus.cc:385] T 98aa6d511d634562b7ff45c12f617bcd P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:36.927934 28498 raft_consensus.cc:740] T 98aa6d511d634562b7ff45c12f617bcd P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cea691cf09a44fda4b6307ef400cbd6, State: Initialized, Role: FOLLOWER
I20260131 03:08:36.927987 28498 consensus_queue.cc:260] T 98aa6d511d634562b7ff45c12f617bcd P 6cea691cf09a44fda4b6307ef400cbd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:36.928103 28498 ts_tablet_manager.cc:1434] T 98aa6d511d634562b7ff45c12f617bcd P 6cea691cf09a44fda4b6307ef400cbd6: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260131 03:08:36.928205 28520 ts_tablet_manager.cc:1950] T f442ed3ece1543318da6d79fd2cf431d P 933d0320a7644b8ebac076ae68a53c3f: Deleting consensus metadata
I20260131 03:08:36.928475 28497 tablet_bootstrap.cc:492] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b: Bootstrap starting.
I20260131 03:08:36.928906 28497 tablet_bootstrap.cc:654] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:36.929445 27966 catalog_manager.cc:4985] TS 933d0320a7644b8ebac076ae68a53c3f (127.19.84.1:33971): tablet f442ed3ece1543318da6d79fd2cf431d (table test-table [id=19fd5cd8ccdd439d8d4dbe1c908b5ec3]) successfully deleted
I20260131 03:08:36.929679 28497 tablet_bootstrap.cc:492] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b: No bootstrap required, opened a new log
I20260131 03:08:36.929733 28497 ts_tablet_manager.cc:1403] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:36.929991 28497 raft_consensus.cc:359] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:36.930053 28497 raft_consensus.cc:385] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:36.930068 28497 raft_consensus.cc:740] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0fb784728cb414594ee1ee6d03e875b, State: Initialized, Role: FOLLOWER
I20260131 03:08:36.930131 28497 consensus_queue.cc:260] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:36.930243 28497 ts_tablet_manager.cc:1434] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:36.930522 28522 ts_tablet_manager.cc:1916] T f442ed3ece1543318da6d79fd2cf431d P d0fb784728cb414594ee1ee6d03e875b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:08:36.931852 28501 tablet_bootstrap.cc:492] T 98aa6d511d634562b7ff45c12f617bcd P 933d0320a7644b8ebac076ae68a53c3f: No bootstrap required, opened a new log
I20260131 03:08:36.931912 28501 ts_tablet_manager.cc:1403] T 98aa6d511d634562b7ff45c12f617bcd P 933d0320a7644b8ebac076ae68a53c3f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260131 03:08:36.932071 28501 raft_consensus.cc:359] T 98aa6d511d634562b7ff45c12f617bcd P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:36.932139 28501 raft_consensus.cc:385] T 98aa6d511d634562b7ff45c12f617bcd P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:36.932160 28501 raft_consensus.cc:740] T 98aa6d511d634562b7ff45c12f617bcd P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 933d0320a7644b8ebac076ae68a53c3f, State: Initialized, Role: FOLLOWER
I20260131 03:08:36.932215 28501 consensus_queue.cc:260] T 98aa6d511d634562b7ff45c12f617bcd P 933d0320a7644b8ebac076ae68a53c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:36.932297 28501 ts_tablet_manager.cc:1434] T 98aa6d511d634562b7ff45c12f617bcd P 933d0320a7644b8ebac076ae68a53c3f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:36.932606 28522 ts_tablet_manager.cc:1929] T f442ed3ece1543318da6d79fd2cf431d P d0fb784728cb414594ee1ee6d03e875b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:08:36.932687 28522 log.cc:1199] T f442ed3ece1543318da6d79fd2cf431d P d0fb784728cb414594ee1ee6d03e875b: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-1/wal/wals/f442ed3ece1543318da6d79fd2cf431d
I20260131 03:08:36.932991 28522 ts_tablet_manager.cc:1950] T f442ed3ece1543318da6d79fd2cf431d P d0fb784728cb414594ee1ee6d03e875b: Deleting consensus metadata
I20260131 03:08:36.933408 27967 catalog_manager.cc:4985] TS d0fb784728cb414594ee1ee6d03e875b (127.19.84.2:38709): tablet f442ed3ece1543318da6d79fd2cf431d (table test-table [id=19fd5cd8ccdd439d8d4dbe1c908b5ec3]) successfully deleted
I20260131 03:08:36.967509 28502 raft_consensus.cc:493] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:08:36.967622 28502 raft_consensus.cc:515] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:36.967943 28502 leader_election.cc:290] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 933d0320a7644b8ebac076ae68a53c3f (127.19.84.1:33971), 6cea691cf09a44fda4b6307ef400cbd6 (127.19.84.3:36069)
I20260131 03:08:36.970623 28220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.968136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:33971 (local address 127.19.84.2:34029)
0131 03:08:36.968198 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:36.968210 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:36.968258 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:36.968499 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:36.968499 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:36.968501 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:36.968723 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.968726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.969640 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.969641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:36.970416 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.970418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.970490 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.970492 (+ 2us) client_negotiation.cc:769] Sending connection context
0131 03:08:36.970523 (+ 31us) client_negotiation.cc:240] Negotiation successful
0131 03:08:36.970552 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260131 03:08:36.970633 28526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.968239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:36069 (local address 127.19.84.2:55589)
0131 03:08:36.968367 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:36.968376 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:36.968425 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:36.968570 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:36.968570 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:36.968572 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:36.968807 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.968809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.969457 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.969458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:36.970313 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:36.970315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.970428 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.970433 (+ 5us) client_negotiation.cc:769] Sending connection context
0131 03:08:36.970471 (+ 38us) client_negotiation.cc:240] Negotiation successful
0131 03:08:36.970531 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260131 03:08:36.971074 28493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.968396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:55589 (local address 127.19.84.3:36069)
0131 03:08:36.968435 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:36.968437 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:36.968446 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:08:36.968475 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:36.968476 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:36.968485 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:36.968555 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:36.968892 (+ 337us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.969355 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.970424 (+ 1069us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.970940 (+ 516us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.970948 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:36.970973 (+ 25us) server_negotiation.cc:300] Negotiation successful
0131 03:08:36.971006 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260131 03:08:36.971443 28423 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98aa6d511d634562b7ff45c12f617bcd" candidate_uuid: "d0fb784728cb414594ee1ee6d03e875b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cea691cf09a44fda4b6307ef400cbd6" is_pre_election: true
I20260131 03:08:36.971521 28496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:36.968207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:34029 (local address 127.19.84.1:33971)
0131 03:08:36.968253 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:36.968255 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:36.968268 (+ 13us) server_negotiation.cc:408] Connection header received
0131 03:08:36.968357 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:36.968358 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:36.968371 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:36.968425 (+ 54us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:36.968817 (+ 392us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.969556 (+ 739us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:36.970681 (+ 1125us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:36.971214 (+ 533us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:36.971221 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:36.971328 (+ 107us) server_negotiation.cc:300] Negotiation successful
0131 03:08:36.971462 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260131 03:08:36.971580 28423 raft_consensus.cc:2468] T 98aa6d511d634562b7ff45c12f617bcd P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0fb784728cb414594ee1ee6d03e875b in term 0.
I20260131 03:08:36.971767 28139 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98aa6d511d634562b7ff45c12f617bcd" candidate_uuid: "d0fb784728cb414594ee1ee6d03e875b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "933d0320a7644b8ebac076ae68a53c3f" is_pre_election: true
I20260131 03:08:36.971856 28139 raft_consensus.cc:2468] T 98aa6d511d634562b7ff45c12f617bcd P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0fb784728cb414594ee1ee6d03e875b in term 0.
I20260131 03:08:36.972010 28214 leader_election.cc:304] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cea691cf09a44fda4b6307ef400cbd6, d0fb784728cb414594ee1ee6d03e875b; no voters:
I20260131 03:08:36.972146 28502 raft_consensus.cc:2804] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:08:36.972190 28502 raft_consensus.cc:493] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:08:36.972211 28502 raft_consensus.cc:3060] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:36.972963 28502 raft_consensus.cc:515] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:36.973109 28502 leader_election.cc:290] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b [CANDIDATE]: Term 1 election: Requested vote from peers 933d0320a7644b8ebac076ae68a53c3f (127.19.84.1:33971), 6cea691cf09a44fda4b6307ef400cbd6 (127.19.84.3:36069)
I20260131 03:08:36.973304 28139 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98aa6d511d634562b7ff45c12f617bcd" candidate_uuid: "d0fb784728cb414594ee1ee6d03e875b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "933d0320a7644b8ebac076ae68a53c3f"
I20260131 03:08:36.973387 28139 raft_consensus.cc:3060] T 98aa6d511d634562b7ff45c12f617bcd P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:36.974167 28139 raft_consensus.cc:2468] T 98aa6d511d634562b7ff45c12f617bcd P 933d0320a7644b8ebac076ae68a53c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0fb784728cb414594ee1ee6d03e875b in term 1.
I20260131 03:08:36.974305 28423 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98aa6d511d634562b7ff45c12f617bcd" candidate_uuid: "d0fb784728cb414594ee1ee6d03e875b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cea691cf09a44fda4b6307ef400cbd6"
I20260131 03:08:36.974381 28423 raft_consensus.cc:3060] T 98aa6d511d634562b7ff45c12f617bcd P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:36.974404 28214 leader_election.cc:304] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 933d0320a7644b8ebac076ae68a53c3f, d0fb784728cb414594ee1ee6d03e875b; no voters:
I20260131 03:08:36.974519 28502 raft_consensus.cc:2804] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:08:36.974678 28502 raft_consensus.cc:697] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b [term 1 LEADER]: Becoming Leader. State: Replica: d0fb784728cb414594ee1ee6d03e875b, State: Running, Role: LEADER
I20260131 03:08:36.974743 28502 consensus_queue.cc:237] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:36.975199 28423 raft_consensus.cc:2468] T 98aa6d511d634562b7ff45c12f617bcd P 6cea691cf09a44fda4b6307ef400cbd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0fb784728cb414594ee1ee6d03e875b in term 1.
I20260131 03:08:36.975468 27980 catalog_manager.cc:5654] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b reported cstate change: term changed from 0 to 1, leader changed from <none> to d0fb784728cb414594ee1ee6d03e875b (127.19.84.2). New cstate: current_term: 1 leader_uuid: "d0fb784728cb414594ee1ee6d03e875b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } health_report { overall_health: UNKNOWN } } }
Jan 31 03:08:37 dist-test-slave-5nnp krb5kdc[27932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828916, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.62@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM
I20260131 03:08:37.011698 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:37.004312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41959 (local address 127.0.0.1:56332)
0131 03:08:37.004496 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:37.004507 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:37.004571 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:37.004737 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:37.004738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:37.005066 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:37.005233 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:37.005236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.006248 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:37.006249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:37.006681 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:37.006683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.006751 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:37.007304 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:37.007308 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:37.008915 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:37.010697 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:37.010704 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:37.010706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:37.010880 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:37.011070 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:37.011071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:37.011071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:37.011097 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:37.011345 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:37.011350 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:37.011446 (+ 96us) client_negotiation.cc:769] Sending connection context
0131 03:08:37.011526 (+ 80us) client_negotiation.cc:240] Negotiation successful
0131 03:08:37.011616 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260131 03:08:37.011813 28040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:37.004392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56332 (local address 127.19.84.62:41959)
0131 03:08:37.004455 (+ 63us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:37.004458 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:37.004581 (+ 123us) server_negotiation.cc:408] Connection header received
0131 03:08:37.004616 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:37.004617 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:37.004630 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:37.004720 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:37.005330 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.006133 (+ 803us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:37.006781 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.006940 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:37.009003 (+ 2063us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:37.009009 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:37.009009 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:37.009017 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:37.010576 (+ 1559us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:37.010971 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:37.010972 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:37.010973 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:37.011001 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:37.011160 (+ 159us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:37.011161 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:37.011162 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:37.011277 (+ 115us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:37.011355 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:37.011583 (+ 228us) server_negotiation.cc:300] Negotiation successful
0131 03:08:37.011695 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260131 03:08:37.013942 27980 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:56332:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260131 03:08:37.014178 27980 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:08:37.015775 28403 tablet_service.cc:1505] Processing CreateTablet for tablet a8a23be1a4aa4d32931765112cf8b87a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=68cd1f9712b24af8b8371a3c9bc7c6e7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260131 03:08:37.015779 28119 tablet_service.cc:1505] Processing CreateTablet for tablet a8a23be1a4aa4d32931765112cf8b87a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=68cd1f9712b24af8b8371a3c9bc7c6e7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260131 03:08:37.015929 28403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8a23be1a4aa4d32931765112cf8b87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:37.015929 28119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8a23be1a4aa4d32931765112cf8b87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:37.017379 28501 tablet_bootstrap.cc:492] T a8a23be1a4aa4d32931765112cf8b87a P 933d0320a7644b8ebac076ae68a53c3f: Bootstrap starting.
I20260131 03:08:37.017421 28261 tablet_service.cc:1505] Processing CreateTablet for tablet a8a23be1a4aa4d32931765112cf8b87a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=68cd1f9712b24af8b8371a3c9bc7c6e7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260131 03:08:37.017552 28261 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8a23be1a4aa4d32931765112cf8b87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:37.017859 28501 tablet_bootstrap.cc:654] T a8a23be1a4aa4d32931765112cf8b87a P 933d0320a7644b8ebac076ae68a53c3f: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:37.018566 28498 tablet_bootstrap.cc:492] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6: Bootstrap starting.
I20260131 03:08:37.018822 28501 tablet_bootstrap.cc:492] T a8a23be1a4aa4d32931765112cf8b87a P 933d0320a7644b8ebac076ae68a53c3f: No bootstrap required, opened a new log
I20260131 03:08:37.018867 28501 ts_tablet_manager.cc:1403] T a8a23be1a4aa4d32931765112cf8b87a P 933d0320a7644b8ebac076ae68a53c3f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:08:37.018955 28498 tablet_bootstrap.cc:654] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:37.019022 28501 raft_consensus.cc:359] T a8a23be1a4aa4d32931765112cf8b87a P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.019085 28501 raft_consensus.cc:385] T a8a23be1a4aa4d32931765112cf8b87a P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:37.019105 28501 raft_consensus.cc:740] T a8a23be1a4aa4d32931765112cf8b87a P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 933d0320a7644b8ebac076ae68a53c3f, State: Initialized, Role: FOLLOWER
I20260131 03:08:37.019155 28501 consensus_queue.cc:260] T a8a23be1a4aa4d32931765112cf8b87a P 933d0320a7644b8ebac076ae68a53c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.019249 28501 ts_tablet_manager.cc:1434] T a8a23be1a4aa4d32931765112cf8b87a P 933d0320a7644b8ebac076ae68a53c3f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260131 03:08:37.019248 28497 tablet_bootstrap.cc:492] T a8a23be1a4aa4d32931765112cf8b87a P d0fb784728cb414594ee1ee6d03e875b: Bootstrap starting.
I20260131 03:08:37.019399 28505 tablet_replica.cc:442] T a8a23be1a4aa4d32931765112cf8b87a P 933d0320a7644b8ebac076ae68a53c3f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } } }
I20260131 03:08:37.019466 28505 tablet_replica.cc:445] T a8a23be1a4aa4d32931765112cf8b87a P 933d0320a7644b8ebac076ae68a53c3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:08:37.019742 28497 tablet_bootstrap.cc:654] T a8a23be1a4aa4d32931765112cf8b87a P d0fb784728cb414594ee1ee6d03e875b: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:37.020663 28497 tablet_bootstrap.cc:492] T a8a23be1a4aa4d32931765112cf8b87a P d0fb784728cb414594ee1ee6d03e875b: No bootstrap required, opened a new log
I20260131 03:08:37.020740 28497 ts_tablet_manager.cc:1403] T a8a23be1a4aa4d32931765112cf8b87a P d0fb784728cb414594ee1ee6d03e875b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260131 03:08:37.020906 28497 raft_consensus.cc:359] T a8a23be1a4aa4d32931765112cf8b87a P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.020975 28497 raft_consensus.cc:385] T a8a23be1a4aa4d32931765112cf8b87a P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:37.021000 28497 raft_consensus.cc:740] T a8a23be1a4aa4d32931765112cf8b87a P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0fb784728cb414594ee1ee6d03e875b, State: Initialized, Role: FOLLOWER
I20260131 03:08:37.021054 28497 consensus_queue.cc:260] T a8a23be1a4aa4d32931765112cf8b87a P d0fb784728cb414594ee1ee6d03e875b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.021166 28497 ts_tablet_manager.cc:1434] T a8a23be1a4aa4d32931765112cf8b87a P d0fb784728cb414594ee1ee6d03e875b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260131 03:08:37.021283 28502 tablet_replica.cc:442] T a8a23be1a4aa4d32931765112cf8b87a P d0fb784728cb414594ee1ee6d03e875b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } } }
I20260131 03:08:37.021371 28502 tablet_replica.cc:445] T a8a23be1a4aa4d32931765112cf8b87a P d0fb784728cb414594ee1ee6d03e875b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:08:37.021848 28498 tablet_bootstrap.cc:492] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6: No bootstrap required, opened a new log
I20260131 03:08:37.021900 28498 ts_tablet_manager.cc:1403] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260131 03:08:37.022080 28498 raft_consensus.cc:359] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.022159 28498 raft_consensus.cc:385] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:37.022182 28498 raft_consensus.cc:740] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cea691cf09a44fda4b6307ef400cbd6, State: Initialized, Role: FOLLOWER
I20260131 03:08:37.022228 28498 consensus_queue.cc:260] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.022322 28498 ts_tablet_manager.cc:1434] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:37.022349 28517 tablet_replica.cc:442] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } } }
I20260131 03:08:37.022419 28517 tablet_replica.cc:445] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:08:37.063585 28517 raft_consensus.cc:493] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:08:37.063706 28517 raft_consensus.cc:515] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.063889 28517 leader_election.cc:290] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 933d0320a7644b8ebac076ae68a53c3f (127.19.84.1:33971), d0fb784728cb414594ee1ee6d03e875b (127.19.84.2:38709)
I20260131 03:08:37.064162 28139 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8a23be1a4aa4d32931765112cf8b87a" candidate_uuid: "6cea691cf09a44fda4b6307ef400cbd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "933d0320a7644b8ebac076ae68a53c3f" is_pre_election: true
I20260131 03:08:37.064210 28281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8a23be1a4aa4d32931765112cf8b87a" candidate_uuid: "6cea691cf09a44fda4b6307ef400cbd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0fb784728cb414594ee1ee6d03e875b" is_pre_election: true
I20260131 03:08:37.064282 28139 raft_consensus.cc:2468] T a8a23be1a4aa4d32931765112cf8b87a P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cea691cf09a44fda4b6307ef400cbd6 in term 0.
I20260131 03:08:37.064313 28281 raft_consensus.cc:2468] T a8a23be1a4aa4d32931765112cf8b87a P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cea691cf09a44fda4b6307ef400cbd6 in term 0.
I20260131 03:08:37.064476 28356 leader_election.cc:304] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cea691cf09a44fda4b6307ef400cbd6, 933d0320a7644b8ebac076ae68a53c3f; no voters:
I20260131 03:08:37.064567 28517 raft_consensus.cc:2804] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:08:37.064606 28517 raft_consensus.cc:493] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:08:37.064664 28517 raft_consensus.cc:3060] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:37.065732 28517 raft_consensus.cc:515] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.065886 28517 leader_election.cc:290] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6 [CANDIDATE]: Term 1 election: Requested vote from peers 933d0320a7644b8ebac076ae68a53c3f (127.19.84.1:33971), d0fb784728cb414594ee1ee6d03e875b (127.19.84.2:38709)
I20260131 03:08:37.066038 28139 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8a23be1a4aa4d32931765112cf8b87a" candidate_uuid: "6cea691cf09a44fda4b6307ef400cbd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "933d0320a7644b8ebac076ae68a53c3f"
I20260131 03:08:37.066111 28139 raft_consensus.cc:3060] T a8a23be1a4aa4d32931765112cf8b87a P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:37.066115 28281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8a23be1a4aa4d32931765112cf8b87a" candidate_uuid: "6cea691cf09a44fda4b6307ef400cbd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0fb784728cb414594ee1ee6d03e875b"
I20260131 03:08:37.066175 28281 raft_consensus.cc:3060] T a8a23be1a4aa4d32931765112cf8b87a P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:37.066829 28281 raft_consensus.cc:2468] T a8a23be1a4aa4d32931765112cf8b87a P d0fb784728cb414594ee1ee6d03e875b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cea691cf09a44fda4b6307ef400cbd6 in term 1.
I20260131 03:08:37.066926 28139 raft_consensus.cc:2468] T a8a23be1a4aa4d32931765112cf8b87a P 933d0320a7644b8ebac076ae68a53c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cea691cf09a44fda4b6307ef400cbd6 in term 1.
I20260131 03:08:37.067035 28357 leader_election.cc:304] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cea691cf09a44fda4b6307ef400cbd6, d0fb784728cb414594ee1ee6d03e875b; no voters:
I20260131 03:08:37.067138 28517 raft_consensus.cc:2804] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:08:37.067206 28517 raft_consensus.cc:697] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6 [term 1 LEADER]: Becoming Leader. State: Replica: 6cea691cf09a44fda4b6307ef400cbd6, State: Running, Role: LEADER
I20260131 03:08:37.067265 28517 consensus_queue.cc:237] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.067615 28503 tablet_replica.cc:442] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6: TxnStatusTablet state changed. Reason: New leader 6cea691cf09a44fda4b6307ef400cbd6. Latest consensus state: current_term: 1 leader_uuid: "6cea691cf09a44fda4b6307ef400cbd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } } }
I20260131 03:08:37.067706 28503 tablet_replica.cc:445] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6: This TxnStatusTablet replica's current role is: LEADER
I20260131 03:08:37.067866 28539 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260131 03:08:37.067968 28539 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260131 03:08:37.067945 27980 catalog_manager.cc:5654] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cea691cf09a44fda4b6307ef400cbd6 (127.19.84.3). New cstate: current_term: 1 leader_uuid: "6cea691cf09a44fda4b6307ef400cbd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } health_report { overall_health: HEALTHY } } }
I20260131 03:08:37.080451 28493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:37.078080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43924 (local address 127.19.84.3:36069)
0131 03:08:37.078121 (+ 41us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:37.078124 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:37.078133 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:08:37.078220 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:37.078221 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:37.078232 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:37.078293 (+ 61us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:08:37.078560 (+ 267us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.079260 (+ 700us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:37.079964 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.080124 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:37.080143 (+ 19us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:08:37.080168 (+ 25us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:08:37.080245 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:37.080348 (+ 103us) server_negotiation.cc:300] Negotiation successful
0131 03:08:37.080372 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260131 03:08:37.080471 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:37.077979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:36069 (local address 127.0.0.1:43924)
0131 03:08:37.078057 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:37.078070 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:37.078126 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:37.078307 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:37.078307 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:37.078309 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:08:37.078459 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:37.078490 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.079351 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:37.079352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:37.079858 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:37.079860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.079966 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:37.079989 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:08:37.080240 (+ 251us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:08:37.080253 (+ 13us) client_negotiation.cc:769] Sending connection context
0131 03:08:37.080343 (+ 90us) client_negotiation.cc:240] Negotiation successful
0131 03:08:37.080396 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260131 03:08:37.081529 28281 raft_consensus.cc:1275] T a8a23be1a4aa4d32931765112cf8b87a P d0fb784728cb414594ee1ee6d03e875b [term 1 FOLLOWER]: Refusing update from remote peer 6cea691cf09a44fda4b6307ef400cbd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:08:37.081565 28139 raft_consensus.cc:1275] T a8a23be1a4aa4d32931765112cf8b87a P 933d0320a7644b8ebac076ae68a53c3f [term 1 FOLLOWER]: Refusing update from remote peer 6cea691cf09a44fda4b6307ef400cbd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:08:37.081837 28503 consensus_queue.cc:1048] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:08:37.081985 28517 consensus_queue.cc:1048] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:08:37.082764 28505 tablet_replica.cc:442] T a8a23be1a4aa4d32931765112cf8b87a P 933d0320a7644b8ebac076ae68a53c3f: TxnStatusTablet state changed. Reason: New leader 6cea691cf09a44fda4b6307ef400cbd6. Latest consensus state: current_term: 1 leader_uuid: "6cea691cf09a44fda4b6307ef400cbd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } } }
I20260131 03:08:37.082854 28505 tablet_replica.cc:445] T a8a23be1a4aa4d32931765112cf8b87a P 933d0320a7644b8ebac076ae68a53c3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:08:37.083842 28502 tablet_replica.cc:442] T a8a23be1a4aa4d32931765112cf8b87a P d0fb784728cb414594ee1ee6d03e875b: TxnStatusTablet state changed. Reason: New leader 6cea691cf09a44fda4b6307ef400cbd6. Latest consensus state: current_term: 1 leader_uuid: "6cea691cf09a44fda4b6307ef400cbd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } } }
I20260131 03:08:37.083906 28502 tablet_replica.cc:445] T a8a23be1a4aa4d32931765112cf8b87a P d0fb784728cb414594ee1ee6d03e875b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:08:37.084174 28517 tablet_replica.cc:442] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cea691cf09a44fda4b6307ef400cbd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } } }
I20260131 03:08:37.084259 28517 tablet_replica.cc:445] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6: This TxnStatusTablet replica's current role is: LEADER
I20260131 03:08:37.084336 28517 tablet_replica.cc:442] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cea691cf09a44fda4b6307ef400cbd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } } }
I20260131 03:08:37.084374 28517 tablet_replica.cc:445] T a8a23be1a4aa4d32931765112cf8b87a P 6cea691cf09a44fda4b6307ef400cbd6: This TxnStatusTablet replica's current role is: LEADER
I20260131 03:08:37.084453 28510 mvcc.cc:204] Tried to move back new op lower bound from 7249219244364169216 to 7249219244307992576. Current Snapshot: MvccSnapshot[applied={T|T < 7249219244364169216 or (T in {7249219244364169216})}]
I20260131 03:08:37.084543 28505 tablet_replica.cc:442] T a8a23be1a4aa4d32931765112cf8b87a P 933d0320a7644b8ebac076ae68a53c3f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cea691cf09a44fda4b6307ef400cbd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } } }
I20260131 03:08:37.084612 28505 tablet_replica.cc:445] T a8a23be1a4aa4d32931765112cf8b87a P 933d0320a7644b8ebac076ae68a53c3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:08:37.085860 28502 tablet_replica.cc:442] T a8a23be1a4aa4d32931765112cf8b87a P d0fb784728cb414594ee1ee6d03e875b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cea691cf09a44fda4b6307ef400cbd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } } }
I20260131 03:08:37.085951 28502 tablet_replica.cc:445] T a8a23be1a4aa4d32931765112cf8b87a P d0fb784728cb414594ee1ee6d03e875b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260131 03:08:37.088681 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:37.086273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43794 (local address 127.19.84.2:38709)
0131 03:08:37.086311 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:37.086312 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:37.086319 (+ 7us) server_negotiation.cc:408] Connection header received
0131 03:08:37.086359 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:37.086360 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:37.086380 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:37.086434 (+ 54us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:08:37.086683 (+ 249us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.087473 (+ 790us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:37.088043 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.088213 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:37.088250 (+ 37us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:08:37.088370 (+ 120us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:08:37.088454 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:37.088583 (+ 129us) server_negotiation.cc:300] Negotiation successful
0131 03:08:37.088610 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260131 03:08:37.093521 28493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:37.091209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:53395 (local address 127.19.84.3:36069)
0131 03:08:37.091276 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:37.091278 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:37.091351 (+ 73us) server_negotiation.cc:408] Connection header received
0131 03:08:37.091468 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:37.091469 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:37.091478 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:37.091534 (+ 56us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:08:37.091854 (+ 320us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.092282 (+ 428us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:37.093097 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.093236 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:37.093255 (+ 19us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:08:37.093278 (+ 23us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:08:37.093338 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:37.093444 (+ 106us) server_negotiation.cc:300] Negotiation successful
0131 03:08:37.093467 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260131 03:08:37.093540 28545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:37.091115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:36069 (local address 127.19.84.2:53395)
0131 03:08:37.091282 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:37.091292 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:37.091354 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:37.091552 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:37.091553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:37.091555 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:08:37.091787 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:37.091788 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.092354 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:37.092355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:37.092996 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:37.092998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.093074 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:37.093090 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:08:37.093356 (+ 266us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:08:37.093358 (+ 2us) client_negotiation.cc:769] Sending connection context
0131 03:08:37.093410 (+ 52us) client_negotiation.cc:240] Negotiation successful
0131 03:08:37.093456 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260131 03:08:37.096144 28423 raft_consensus.cc:1275] T 98aa6d511d634562b7ff45c12f617bcd P 6cea691cf09a44fda4b6307ef400cbd6 [term 1 FOLLOWER]: Refusing update from remote peer d0fb784728cb414594ee1ee6d03e875b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:08:37.096151 28139 raft_consensus.cc:1275] T 98aa6d511d634562b7ff45c12f617bcd P 933d0320a7644b8ebac076ae68a53c3f [term 1 FOLLOWER]: Refusing update from remote peer d0fb784728cb414594ee1ee6d03e875b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:08:37.096439 28527 consensus_queue.cc:1048] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b [LEADER]: Connected to new peer: Peer: permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:08:37.096573 28502 consensus_queue.cc:1048] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:08:37.098495 28512 mvcc.cc:204] Tried to move back new op lower bound from 7249219244423405568 to 7249219243929321472. Current Snapshot: MvccSnapshot[applied={T|T < 7249219244423405568}]
I20260131 03:08:37.104065 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:37.102084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:52715 (local address 127.19.84.2:38709)
0131 03:08:37.102183 (+ 99us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:37.102185 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:37.102193 (+ 8us) server_negotiation.cc:408] Connection header received
0131 03:08:37.102212 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:37.102213 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:37.102223 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:37.102285 (+ 62us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:08:37.102499 (+ 214us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.102952 (+ 453us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:37.103583 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.103737 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:37.103754 (+ 17us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:08:37.103774 (+ 20us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:08:37.103841 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:37.103979 (+ 138us) server_negotiation.cc:300] Negotiation successful
0131 03:08:37.104001 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260131 03:08:37.104089 28475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:37.101991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:38709 (local address 127.19.84.3:52715)
0131 03:08:37.102069 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:37.102081 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:37.102119 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:37.102286 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:37.102287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:37.102288 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:08:37.102424 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:37.102425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.103050 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:37.103052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:37.103507 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:37.103509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.103586 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:37.103603 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:08:37.103910 (+ 307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:08:37.103912 (+ 2us) client_negotiation.cc:769] Sending connection context
0131 03:08:37.103974 (+ 62us) client_negotiation.cc:240] Negotiation successful
0131 03:08:37.104022 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260131 03:08:37.114485 27979 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:08:37.114604 27979 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:08:37.115243 27979 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1: Sending DeleteTablet for 3 replicas of tablet 98aa6d511d634562b7ff45c12f617bcd
I20260131 03:08:37.115512 28119 tablet_service.cc:1552] Processing DeleteTablet for tablet 98aa6d511d634562b7ff45c12f617bcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:37 UTC) from {username='oryx', principal='oryx/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:45880
I20260131 03:08:37.115628 28520 tablet_replica.cc:333] T 98aa6d511d634562b7ff45c12f617bcd P 933d0320a7644b8ebac076ae68a53c3f: stopping tablet replica
I20260131 03:08:37.115726 28520 raft_consensus.cc:2243] T 98aa6d511d634562b7ff45c12f617bcd P 933d0320a7644b8ebac076ae68a53c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:08:37.115775 28520 raft_consensus.cc:2272] T 98aa6d511d634562b7ff45c12f617bcd P 933d0320a7644b8ebac076ae68a53c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:08:37.115850 28261 tablet_service.cc:1552] Processing DeleteTablet for tablet 98aa6d511d634562b7ff45c12f617bcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:37 UTC) from {username='oryx', principal='oryx/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:43784
I20260131 03:08:37.116003 28522 tablet_replica.cc:333] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b: stopping tablet replica
I20260131 03:08:37.116052 28522 raft_consensus.cc:2243] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:08:37.116130 28522 raft_consensus.cc:2272] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:08:37.116273 28403 tablet_service.cc:1552] Processing DeleteTablet for tablet 98aa6d511d634562b7ff45c12f617bcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:37 UTC) from {username='oryx', principal='oryx/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:43906
I20260131 03:08:37.116360 28521 tablet_replica.cc:333] T 98aa6d511d634562b7ff45c12f617bcd P 6cea691cf09a44fda4b6307ef400cbd6: stopping tablet replica
I20260131 03:08:37.116423 28521 raft_consensus.cc:2243] T 98aa6d511d634562b7ff45c12f617bcd P 6cea691cf09a44fda4b6307ef400cbd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:08:37.116495 28521 raft_consensus.cc:2272] T 98aa6d511d634562b7ff45c12f617bcd P 6cea691cf09a44fda4b6307ef400cbd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:08:37.116923 28521 ts_tablet_manager.cc:1916] T 98aa6d511d634562b7ff45c12f617bcd P 6cea691cf09a44fda4b6307ef400cbd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:08:37.118507 28520 ts_tablet_manager.cc:1916] T 98aa6d511d634562b7ff45c12f617bcd P 933d0320a7644b8ebac076ae68a53c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:08:37.118592 28521 ts_tablet_manager.cc:1929] T 98aa6d511d634562b7ff45c12f617bcd P 6cea691cf09a44fda4b6307ef400cbd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:08:37.118646 28521 log.cc:1199] T 98aa6d511d634562b7ff45c12f617bcd P 6cea691cf09a44fda4b6307ef400cbd6: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/98aa6d511d634562b7ff45c12f617bcd
I20260131 03:08:37.118886 28521 ts_tablet_manager.cc:1950] T 98aa6d511d634562b7ff45c12f617bcd P 6cea691cf09a44fda4b6307ef400cbd6: Deleting consensus metadata
I20260131 03:08:37.119989 28520 ts_tablet_manager.cc:1929] T 98aa6d511d634562b7ff45c12f617bcd P 933d0320a7644b8ebac076ae68a53c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:08:37.120052 28520 log.cc:1199] T 98aa6d511d634562b7ff45c12f617bcd P 933d0320a7644b8ebac076ae68a53c3f: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-0/wal/wals/98aa6d511d634562b7ff45c12f617bcd
I20260131 03:08:37.120268 28520 ts_tablet_manager.cc:1950] T 98aa6d511d634562b7ff45c12f617bcd P 933d0320a7644b8ebac076ae68a53c3f: Deleting consensus metadata
I20260131 03:08:37.120491 27966 catalog_manager.cc:4985] TS 6cea691cf09a44fda4b6307ef400cbd6 (127.19.84.3:36069): tablet 98aa6d511d634562b7ff45c12f617bcd (table test-table [id=3f4f68e88485428abd2341b997041a6a]) successfully deleted
I20260131 03:08:37.120684 27966 catalog_manager.cc:4985] TS 933d0320a7644b8ebac076ae68a53c3f (127.19.84.1:33971): tablet 98aa6d511d634562b7ff45c12f617bcd (table test-table [id=3f4f68e88485428abd2341b997041a6a]) successfully deleted
I20260131 03:08:37.120868 28522 ts_tablet_manager.cc:1916] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:08:37.122259 28522 ts_tablet_manager.cc:1929] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:08:37.122320 28522 log.cc:1199] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-1/wal/wals/98aa6d511d634562b7ff45c12f617bcd
I20260131 03:08:37.122522 28522 ts_tablet_manager.cc:1950] T 98aa6d511d634562b7ff45c12f617bcd P d0fb784728cb414594ee1ee6d03e875b: Deleting consensus metadata
I20260131 03:08:37.122989 27967 catalog_manager.cc:4985] TS d0fb784728cb414594ee1ee6d03e875b (127.19.84.2:38709): tablet 98aa6d511d634562b7ff45c12f617bcd (table test-table [id=3f4f68e88485428abd2341b997041a6a]) successfully deleted
Jan 31 03:08:37 dist-test-slave-5nnp krb5kdc[27932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 31 03:08:37 dist-test-slave-5nnp krb5kdc[27932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:37 dist-test-slave-5nnp krb5kdc[27932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
W20260131 03:08:37.150436 28040 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:37.144117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56346 (local address 127.19.84.62:41959)
0131 03:08:37.144164 (+ 47us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:37.144166 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:37.144329 (+ 163us) server_negotiation.cc:408] Connection header received
0131 03:08:37.144388 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:37.144390 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:37.144403 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:37.144468 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:37.145424 (+ 956us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.146243 (+ 819us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:37.147069 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.147253 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:37.150368 (+ 3115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56346: BlockingRecv error: recv got EOF from 127.0.0.1:56346 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jan 31 03:08:37 dist-test-slave-5nnp krb5kdc[27932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM
I20260131 03:08:37.163622 28040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:37.151499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56358 (local address 127.19.84.62:41959)
0131 03:08:37.151542 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:37.151543 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:37.155567 (+ 4024us) server_negotiation.cc:408] Connection header received
0131 03:08:37.155635 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:37.155636 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:37.155647 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:37.155709 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:37.156470 (+ 761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.157235 (+ 765us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:37.158032 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.158204 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:37.160620 (+ 2416us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:37.160624 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:37.160624 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:37.160631 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:37.162346 (+ 1715us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:37.162744 (+ 398us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:37.162745 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:37.162745 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:37.162770 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:37.162943 (+ 173us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:37.162944 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:37.162944 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:37.163059 (+ 115us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:37.163156 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:37.163298 (+ 142us) server_negotiation.cc:300] Negotiation successful
0131 03:08:37.163399 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260131 03:08:37.165428 27979 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:08:37.165596 27979 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:08:37.166940 28261 tablet_service.cc:1505] Processing CreateTablet for tablet 6c9e5ef100634cb0babd9eea4456e18f (DEFAULT_TABLE table=test-table [id=1c8c0ede5d694ac7bc046e1f172a4126]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:08:37.167068 28261 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c9e5ef100634cb0babd9eea4456e18f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:37.167068 28119 tablet_service.cc:1505] Processing CreateTablet for tablet 6c9e5ef100634cb0babd9eea4456e18f (DEFAULT_TABLE table=test-table [id=1c8c0ede5d694ac7bc046e1f172a4126]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:08:37.167194 28119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c9e5ef100634cb0babd9eea4456e18f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:37.168467 28403 tablet_service.cc:1505] Processing CreateTablet for tablet 6c9e5ef100634cb0babd9eea4456e18f (DEFAULT_TABLE table=test-table [id=1c8c0ede5d694ac7bc046e1f172a4126]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:08:37.168579 28403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c9e5ef100634cb0babd9eea4456e18f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:37.169799 28498 tablet_bootstrap.cc:492] T 6c9e5ef100634cb0babd9eea4456e18f P 6cea691cf09a44fda4b6307ef400cbd6: Bootstrap starting.
I20260131 03:08:37.169807 28497 tablet_bootstrap.cc:492] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b: Bootstrap starting.
I20260131 03:08:37.170138 28498 tablet_bootstrap.cc:654] T 6c9e5ef100634cb0babd9eea4456e18f P 6cea691cf09a44fda4b6307ef400cbd6: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:37.170431 28497 tablet_bootstrap.cc:654] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:37.170747 28501 tablet_bootstrap.cc:492] T 6c9e5ef100634cb0babd9eea4456e18f P 933d0320a7644b8ebac076ae68a53c3f: Bootstrap starting.
I20260131 03:08:37.171072 28497 tablet_bootstrap.cc:492] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b: No bootstrap required, opened a new log
I20260131 03:08:37.171114 28497 ts_tablet_manager.cc:1403] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260131 03:08:37.171180 28501 tablet_bootstrap.cc:654] T 6c9e5ef100634cb0babd9eea4456e18f P 933d0320a7644b8ebac076ae68a53c3f: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:37.171258 28497 raft_consensus.cc:359] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.171329 28497 raft_consensus.cc:385] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:37.171350 28497 raft_consensus.cc:740] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0fb784728cb414594ee1ee6d03e875b, State: Initialized, Role: FOLLOWER
I20260131 03:08:37.171422 28497 consensus_queue.cc:260] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.171516 28497 ts_tablet_manager.cc:1434] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:37.172552 28501 tablet_bootstrap.cc:492] T 6c9e5ef100634cb0babd9eea4456e18f P 933d0320a7644b8ebac076ae68a53c3f: No bootstrap required, opened a new log
I20260131 03:08:37.172601 28501 ts_tablet_manager.cc:1403] T 6c9e5ef100634cb0babd9eea4456e18f P 933d0320a7644b8ebac076ae68a53c3f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260131 03:08:37.172755 28501 raft_consensus.cc:359] T 6c9e5ef100634cb0babd9eea4456e18f P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.172812 28501 raft_consensus.cc:385] T 6c9e5ef100634cb0babd9eea4456e18f P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:37.172832 28501 raft_consensus.cc:740] T 6c9e5ef100634cb0babd9eea4456e18f P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 933d0320a7644b8ebac076ae68a53c3f, State: Initialized, Role: FOLLOWER
I20260131 03:08:37.172878 28501 consensus_queue.cc:260] T 6c9e5ef100634cb0babd9eea4456e18f P 933d0320a7644b8ebac076ae68a53c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.172974 28501 ts_tablet_manager.cc:1434] T 6c9e5ef100634cb0babd9eea4456e18f P 933d0320a7644b8ebac076ae68a53c3f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:37.174276 28498 tablet_bootstrap.cc:492] T 6c9e5ef100634cb0babd9eea4456e18f P 6cea691cf09a44fda4b6307ef400cbd6: No bootstrap required, opened a new log
I20260131 03:08:37.174337 28498 ts_tablet_manager.cc:1403] T 6c9e5ef100634cb0babd9eea4456e18f P 6cea691cf09a44fda4b6307ef400cbd6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260131 03:08:37.174458 28498 raft_consensus.cc:359] T 6c9e5ef100634cb0babd9eea4456e18f P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.174526 28498 raft_consensus.cc:385] T 6c9e5ef100634cb0babd9eea4456e18f P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:37.174547 28498 raft_consensus.cc:740] T 6c9e5ef100634cb0babd9eea4456e18f P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cea691cf09a44fda4b6307ef400cbd6, State: Initialized, Role: FOLLOWER
I20260131 03:08:37.174597 28498 consensus_queue.cc:260] T 6c9e5ef100634cb0babd9eea4456e18f P 6cea691cf09a44fda4b6307ef400cbd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.174685 28498 ts_tablet_manager.cc:1434] T 6c9e5ef100634cb0babd9eea4456e18f P 6cea691cf09a44fda4b6307ef400cbd6: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260131 03:08:37.251500 28548 raft_consensus.cc:493] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:08:37.251649 28548 raft_consensus.cc:515] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.251816 28548 leader_election.cc:290] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 933d0320a7644b8ebac076ae68a53c3f (127.19.84.1:33971), 6cea691cf09a44fda4b6307ef400cbd6 (127.19.84.3:36069)
I20260131 03:08:37.252031 28139 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c9e5ef100634cb0babd9eea4456e18f" candidate_uuid: "d0fb784728cb414594ee1ee6d03e875b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "933d0320a7644b8ebac076ae68a53c3f" is_pre_election: true
I20260131 03:08:37.252046 28423 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c9e5ef100634cb0babd9eea4456e18f" candidate_uuid: "d0fb784728cb414594ee1ee6d03e875b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cea691cf09a44fda4b6307ef400cbd6" is_pre_election: true
I20260131 03:08:37.252120 28139 raft_consensus.cc:2468] T 6c9e5ef100634cb0babd9eea4456e18f P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0fb784728cb414594ee1ee6d03e875b in term 0.
I20260131 03:08:37.252141 28423 raft_consensus.cc:2468] T 6c9e5ef100634cb0babd9eea4456e18f P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0fb784728cb414594ee1ee6d03e875b in term 0.
I20260131 03:08:37.252276 28214 leader_election.cc:304] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 933d0320a7644b8ebac076ae68a53c3f, d0fb784728cb414594ee1ee6d03e875b; no voters:
I20260131 03:08:37.252391 28548 raft_consensus.cc:2804] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:08:37.252444 28548 raft_consensus.cc:493] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:08:37.252465 28548 raft_consensus.cc:3060] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:37.253715 28548 raft_consensus.cc:515] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.253849 28548 leader_election.cc:290] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b [CANDIDATE]: Term 1 election: Requested vote from peers 933d0320a7644b8ebac076ae68a53c3f (127.19.84.1:33971), 6cea691cf09a44fda4b6307ef400cbd6 (127.19.84.3:36069)
I20260131 03:08:37.254020 28139 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c9e5ef100634cb0babd9eea4456e18f" candidate_uuid: "d0fb784728cb414594ee1ee6d03e875b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "933d0320a7644b8ebac076ae68a53c3f"
I20260131 03:08:37.254092 28139 raft_consensus.cc:3060] T 6c9e5ef100634cb0babd9eea4456e18f P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:37.254065 28423 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c9e5ef100634cb0babd9eea4456e18f" candidate_uuid: "d0fb784728cb414594ee1ee6d03e875b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cea691cf09a44fda4b6307ef400cbd6"
I20260131 03:08:37.254135 28423 raft_consensus.cc:3060] T 6c9e5ef100634cb0babd9eea4456e18f P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:37.254870 28139 raft_consensus.cc:2468] T 6c9e5ef100634cb0babd9eea4456e18f P 933d0320a7644b8ebac076ae68a53c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0fb784728cb414594ee1ee6d03e875b in term 1.
I20260131 03:08:37.255066 28423 raft_consensus.cc:2468] T 6c9e5ef100634cb0babd9eea4456e18f P 6cea691cf09a44fda4b6307ef400cbd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0fb784728cb414594ee1ee6d03e875b in term 1.
I20260131 03:08:37.255241 28214 leader_election.cc:304] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cea691cf09a44fda4b6307ef400cbd6, d0fb784728cb414594ee1ee6d03e875b; no voters:
I20260131 03:08:37.255347 28548 raft_consensus.cc:2804] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:08:37.255424 28548 raft_consensus.cc:697] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b [term 1 LEADER]: Becoming Leader. State: Replica: d0fb784728cb414594ee1ee6d03e875b, State: Running, Role: LEADER
I20260131 03:08:37.255475 28548 consensus_queue.cc:237] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.256067 27979 catalog_manager.cc:5654] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b reported cstate change: term changed from 0 to 1, leader changed from <none> to d0fb784728cb414594ee1ee6d03e875b (127.19.84.2). New cstate: current_term: 1 leader_uuid: "d0fb784728cb414594ee1ee6d03e875b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } health_report { overall_health: UNKNOWN } } }
I20260131 03:08:37.270300 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:37.268101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43800 (local address 127.19.84.2:38709)
0131 03:08:37.268144 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:37.268146 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:37.268182 (+ 36us) server_negotiation.cc:408] Connection header received
0131 03:08:37.268206 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:37.268207 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:37.268216 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:37.268258 (+ 42us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:08:37.268597 (+ 339us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.269043 (+ 446us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:37.269851 (+ 808us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.269984 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:37.270001 (+ 17us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:08:37.270024 (+ 23us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:08:37.270080 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:37.270171 (+ 91us) server_negotiation.cc:300] Negotiation successful
0131 03:08:37.270193 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260131 03:08:37.271121 28423 raft_consensus.cc:1275] T 6c9e5ef100634cb0babd9eea4456e18f P 6cea691cf09a44fda4b6307ef400cbd6 [term 1 FOLLOWER]: Refusing update from remote peer d0fb784728cb414594ee1ee6d03e875b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:08:37.271144 28139 raft_consensus.cc:1275] T 6c9e5ef100634cb0babd9eea4456e18f P 933d0320a7644b8ebac076ae68a53c3f [term 1 FOLLOWER]: Refusing update from remote peer d0fb784728cb414594ee1ee6d03e875b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:08:37.271474 28548 consensus_queue.cc:1048] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b [LEADER]: Connected to new peer: Peer: permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:08:37.271598 28527 consensus_queue.cc:1048] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:08:37.274114 27978 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:08:37.274219 27978 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:08:37.275272 27978 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1: Sending DeleteTablet for 3 replicas of tablet 6c9e5ef100634cb0babd9eea4456e18f
I20260131 03:08:37.275571 28119 tablet_service.cc:1552] Processing DeleteTablet for tablet 6c9e5ef100634cb0babd9eea4456e18f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:37 UTC) from {username='oryx', principal='oryx/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:45880
I20260131 03:08:37.275700 28520 tablet_replica.cc:333] T 6c9e5ef100634cb0babd9eea4456e18f P 933d0320a7644b8ebac076ae68a53c3f: stopping tablet replica
I20260131 03:08:37.275758 28520 raft_consensus.cc:2243] T 6c9e5ef100634cb0babd9eea4456e18f P 933d0320a7644b8ebac076ae68a53c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:08:37.275748 28403 tablet_service.cc:1552] Processing DeleteTablet for tablet 6c9e5ef100634cb0babd9eea4456e18f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:37 UTC) from {username='oryx', principal='oryx/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:43906
I20260131 03:08:37.275812 28520 raft_consensus.cc:2272] T 6c9e5ef100634cb0babd9eea4456e18f P 933d0320a7644b8ebac076ae68a53c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:08:37.275841 28521 tablet_replica.cc:333] T 6c9e5ef100634cb0babd9eea4456e18f P 6cea691cf09a44fda4b6307ef400cbd6: stopping tablet replica
I20260131 03:08:37.275892 28521 raft_consensus.cc:2243] T 6c9e5ef100634cb0babd9eea4456e18f P 6cea691cf09a44fda4b6307ef400cbd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:08:37.275954 28521 raft_consensus.cc:2272] T 6c9e5ef100634cb0babd9eea4456e18f P 6cea691cf09a44fda4b6307ef400cbd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:08:37.276149 28520 ts_tablet_manager.cc:1916] T 6c9e5ef100634cb0babd9eea4456e18f P 933d0320a7644b8ebac076ae68a53c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:08:37.276190 28119 tablet_service.cc:1552] Processing DeleteTablet for tablet 6c9e5ef100634cb0babd9eea4456e18f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:37 UTC) from {username='oryx', principal='oryx/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:45880
I20260131 03:08:37.276245 28521 ts_tablet_manager.cc:1916] T 6c9e5ef100634cb0babd9eea4456e18f P 6cea691cf09a44fda4b6307ef400cbd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:08:37.276794 28261 tablet_service.cc:1552] Processing DeleteTablet for tablet 6c9e5ef100634cb0babd9eea4456e18f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:37 UTC) from {username='oryx', principal='oryx/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:43784
I20260131 03:08:37.276887 28260 tablet_service.cc:1552] Processing DeleteTablet for tablet 6c9e5ef100634cb0babd9eea4456e18f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:37 UTC) from {username='oryx', principal='oryx/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:43784
I20260131 03:08:37.276940 28522 tablet_replica.cc:333] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b: stopping tablet replica
I20260131 03:08:37.276973 28522 raft_consensus.cc:2243] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:08:37.277041 28522 raft_consensus.cc:2272] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:08:37.277499 28522 ts_tablet_manager.cc:1916] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:08:37.277547 27980 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:08:37.277690 27980 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:08:37.278101 28521 ts_tablet_manager.cc:1929] T 6c9e5ef100634cb0babd9eea4456e18f P 6cea691cf09a44fda4b6307ef400cbd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:08:37.278159 28521 log.cc:1199] T 6c9e5ef100634cb0babd9eea4456e18f P 6cea691cf09a44fda4b6307ef400cbd6: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/6c9e5ef100634cb0babd9eea4456e18f
I20260131 03:08:37.278373 28521 ts_tablet_manager.cc:1950] T 6c9e5ef100634cb0babd9eea4456e18f P 6cea691cf09a44fda4b6307ef400cbd6: Deleting consensus metadata
I20260131 03:08:37.278839 28119 tablet_service.cc:1505] Processing CreateTablet for tablet 051b8e1ad58e497aa4950c834b54095b (DEFAULT_TABLE table=test-table [id=ebe905ad7e654f89ae6f9f82e57120d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:08:37.278870 28403 tablet_service.cc:1505] Processing CreateTablet for tablet 051b8e1ad58e497aa4950c834b54095b (DEFAULT_TABLE table=test-table [id=ebe905ad7e654f89ae6f9f82e57120d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:08:37.278950 27966 catalog_manager.cc:4985] TS 6cea691cf09a44fda4b6307ef400cbd6 (127.19.84.3:36069): tablet 6c9e5ef100634cb0babd9eea4456e18f (table test-table [id=1c8c0ede5d694ac7bc046e1f172a4126]) successfully deleted
I20260131 03:08:37.278970 28403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 051b8e1ad58e497aa4950c834b54095b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:37.278983 28522 ts_tablet_manager.cc:1929] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:08:37.279027 28522 log.cc:1199] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-1/wal/wals/6c9e5ef100634cb0babd9eea4456e18f
I20260131 03:08:37.279261 28522 ts_tablet_manager.cc:1950] T 6c9e5ef100634cb0babd9eea4456e18f P d0fb784728cb414594ee1ee6d03e875b: Deleting consensus metadata
I20260131 03:08:37.279465 28260 tablet_service.cc:1505] Processing CreateTablet for tablet 051b8e1ad58e497aa4950c834b54095b (DEFAULT_TABLE table=test-table [id=ebe905ad7e654f89ae6f9f82e57120d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:08:37.279554 28260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 051b8e1ad58e497aa4950c834b54095b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:37.279635 28520 ts_tablet_manager.cc:1929] T 6c9e5ef100634cb0babd9eea4456e18f P 933d0320a7644b8ebac076ae68a53c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:08:37.279688 28520 log.cc:1199] T 6c9e5ef100634cb0babd9eea4456e18f P 933d0320a7644b8ebac076ae68a53c3f: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-0/wal/wals/6c9e5ef100634cb0babd9eea4456e18f
I20260131 03:08:37.279901 28520 ts_tablet_manager.cc:1950] T 6c9e5ef100634cb0babd9eea4456e18f P 933d0320a7644b8ebac076ae68a53c3f: Deleting consensus metadata
I20260131 03:08:37.280022 28119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 051b8e1ad58e497aa4950c834b54095b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:37.280287 28498 tablet_bootstrap.cc:492] T 051b8e1ad58e497aa4950c834b54095b P 6cea691cf09a44fda4b6307ef400cbd6: Bootstrap starting.
I20260131 03:08:37.280596 28498 tablet_bootstrap.cc:654] T 051b8e1ad58e497aa4950c834b54095b P 6cea691cf09a44fda4b6307ef400cbd6: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:37.281183 28498 tablet_bootstrap.cc:492] T 051b8e1ad58e497aa4950c834b54095b P 6cea691cf09a44fda4b6307ef400cbd6: No bootstrap required, opened a new log
I20260131 03:08:37.281185 28497 tablet_bootstrap.cc:492] T 051b8e1ad58e497aa4950c834b54095b P d0fb784728cb414594ee1ee6d03e875b: Bootstrap starting.
I20260131 03:08:37.281236 28498 ts_tablet_manager.cc:1403] T 051b8e1ad58e497aa4950c834b54095b P 6cea691cf09a44fda4b6307ef400cbd6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260131 03:08:37.281358 28498 raft_consensus.cc:359] T 051b8e1ad58e497aa4950c834b54095b P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.281409 28501 tablet_bootstrap.cc:492] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f: Bootstrap starting.
I20260131 03:08:37.281424 28498 raft_consensus.cc:385] T 051b8e1ad58e497aa4950c834b54095b P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:37.281447 28498 raft_consensus.cc:740] T 051b8e1ad58e497aa4950c834b54095b P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cea691cf09a44fda4b6307ef400cbd6, State: Initialized, Role: FOLLOWER
I20260131 03:08:37.281495 28498 consensus_queue.cc:260] T 051b8e1ad58e497aa4950c834b54095b P 6cea691cf09a44fda4b6307ef400cbd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.281512 28497 tablet_bootstrap.cc:654] T 051b8e1ad58e497aa4950c834b54095b P d0fb784728cb414594ee1ee6d03e875b: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:37.281599 28498 ts_tablet_manager.cc:1434] T 051b8e1ad58e497aa4950c834b54095b P 6cea691cf09a44fda4b6307ef400cbd6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:37.281857 27966 catalog_manager.cc:4985] TS 933d0320a7644b8ebac076ae68a53c3f (127.19.84.1:33971): tablet 6c9e5ef100634cb0babd9eea4456e18f (table test-table [id=1c8c0ede5d694ac7bc046e1f172a4126]) successfully deleted
I20260131 03:08:37.281957 28501 tablet_bootstrap.cc:654] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f: Neither blocks nor log segments found. Creating new log.
W20260131 03:08:37.281975 27966 catalog_manager.cc:4952] TS 933d0320a7644b8ebac076ae68a53c3f (127.19.84.1:33971): delete failed for tablet 6c9e5ef100634cb0babd9eea4456e18f because the tablet was not found. No further retry: Not found: Tablet not found: 6c9e5ef100634cb0babd9eea4456e18f
I20260131 03:08:37.282016 27967 catalog_manager.cc:4985] TS d0fb784728cb414594ee1ee6d03e875b (127.19.84.2:38709): tablet 6c9e5ef100634cb0babd9eea4456e18f (table test-table [id=1c8c0ede5d694ac7bc046e1f172a4126]) successfully deleted
W20260131 03:08:37.282076 27967 catalog_manager.cc:4952] TS d0fb784728cb414594ee1ee6d03e875b (127.19.84.2:38709): delete failed for tablet 6c9e5ef100634cb0babd9eea4456e18f because the tablet was not found. No further retry: Not found: Tablet not found: 6c9e5ef100634cb0babd9eea4456e18f
I20260131 03:08:37.282363 28497 tablet_bootstrap.cc:492] T 051b8e1ad58e497aa4950c834b54095b P d0fb784728cb414594ee1ee6d03e875b: No bootstrap required, opened a new log
I20260131 03:08:37.282415 28497 ts_tablet_manager.cc:1403] T 051b8e1ad58e497aa4950c834b54095b P d0fb784728cb414594ee1ee6d03e875b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260131 03:08:37.282546 28497 raft_consensus.cc:359] T 051b8e1ad58e497aa4950c834b54095b P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.282619 28497 raft_consensus.cc:385] T 051b8e1ad58e497aa4950c834b54095b P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:37.282642 28497 raft_consensus.cc:740] T 051b8e1ad58e497aa4950c834b54095b P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0fb784728cb414594ee1ee6d03e875b, State: Initialized, Role: FOLLOWER
I20260131 03:08:37.282809 28497 consensus_queue.cc:260] T 051b8e1ad58e497aa4950c834b54095b P d0fb784728cb414594ee1ee6d03e875b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.282872 28501 tablet_bootstrap.cc:492] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f: No bootstrap required, opened a new log
I20260131 03:08:37.282912 28501 ts_tablet_manager.cc:1403] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260131 03:08:37.282929 28497 ts_tablet_manager.cc:1434] T 051b8e1ad58e497aa4950c834b54095b P d0fb784728cb414594ee1ee6d03e875b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:37.283049 28501 raft_consensus.cc:359] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.283141 28501 raft_consensus.cc:385] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:37.283161 28501 raft_consensus.cc:740] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 933d0320a7644b8ebac076ae68a53c3f, State: Initialized, Role: FOLLOWER
I20260131 03:08:37.283205 28501 consensus_queue.cc:260] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.283572 28501 ts_tablet_manager.cc:1434] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260131 03:08:37.423481 28470 tablet.cc:2378] T 051b8e1ad58e497aa4950c834b54095b P 6cea691cf09a44fda4b6307ef400cbd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:08:37.449622 28505 raft_consensus.cc:493] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:08:37.449728 28505 raft_consensus.cc:515] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.450088 28505 leader_election.cc:290] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0fb784728cb414594ee1ee6d03e875b (127.19.84.2:38709), 6cea691cf09a44fda4b6307ef400cbd6 (127.19.84.3:36069)
I20260131 03:08:37.452467 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:37.450286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:36069 (local address 127.19.84.1:49203)
0131 03:08:37.450436 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:37.450448 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:37.450518 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:37.450683 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:37.450684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:37.450687 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:37.450875 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:37.450877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.451607 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:37.451608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:37.452196 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:37.452198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.452304 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:37.452309 (+ 5us) client_negotiation.cc:769] Sending connection context
0131 03:08:37.452343 (+ 34us) client_negotiation.cc:240] Negotiation successful
0131 03:08:37.452381 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":64,"threads_started":1}
I20260131 03:08:37.452601 28575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:37.450359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:38709 (local address 127.19.84.1:42861)
0131 03:08:37.450527 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:37.450542 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:37.450620 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:37.450785 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:37.450786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:37.450787 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:08:37.451037 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:37.451039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.451778 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:37.451779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:37.452357 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:37.452359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.452445 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:37.452457 (+ 12us) client_negotiation.cc:769] Sending connection context
0131 03:08:37.452491 (+ 34us) client_negotiation.cc:240] Negotiation successful
0131 03:08:37.452525 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":106,"threads_started":1}
I20260131 03:08:37.452849 28493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:37.450327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:49203 (local address 127.19.84.3:36069)
0131 03:08:37.450390 (+ 63us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:37.450393 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:37.450515 (+ 122us) server_negotiation.cc:408] Connection header received
0131 03:08:37.450590 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:37.450592 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:37.450610 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:37.450678 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:37.450958 (+ 280us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.451495 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:37.452283 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.452726 (+ 443us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:37.452736 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:37.452760 (+ 24us) server_negotiation.cc:300] Negotiation successful
0131 03:08:37.452794 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260131 03:08:37.453053 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:37.450414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:42861 (local address 127.19.84.2:38709)
0131 03:08:37.450481 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:37.450484 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:37.450603 (+ 119us) server_negotiation.cc:408] Connection header received
0131 03:08:37.450686 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:37.450687 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:37.450696 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:37.450766 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:08:37.451116 (+ 350us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.451690 (+ 574us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:37.452457 (+ 767us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.452936 (+ 479us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:37.452943 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:37.452966 (+ 23us) server_negotiation.cc:300] Negotiation successful
0131 03:08:37.452994 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260131 03:08:37.453189 28423 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "051b8e1ad58e497aa4950c834b54095b" candidate_uuid: "933d0320a7644b8ebac076ae68a53c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cea691cf09a44fda4b6307ef400cbd6" is_pre_election: true
I20260131 03:08:37.453220 28281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "051b8e1ad58e497aa4950c834b54095b" candidate_uuid: "933d0320a7644b8ebac076ae68a53c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0fb784728cb414594ee1ee6d03e875b" is_pre_election: true
I20260131 03:08:37.453287 28423 raft_consensus.cc:2468] T 051b8e1ad58e497aa4950c834b54095b P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 933d0320a7644b8ebac076ae68a53c3f in term 0.
I20260131 03:08:37.453290 28281 raft_consensus.cc:2468] T 051b8e1ad58e497aa4950c834b54095b P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 933d0320a7644b8ebac076ae68a53c3f in term 0.
I20260131 03:08:37.453476 28072 leader_election.cc:304] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cea691cf09a44fda4b6307ef400cbd6, 933d0320a7644b8ebac076ae68a53c3f; no voters:
I20260131 03:08:37.453608 28505 raft_consensus.cc:2804] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:08:37.453657 28505 raft_consensus.cc:493] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:08:37.453678 28505 raft_consensus.cc:3060] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:37.454267 28505 raft_consensus.cc:515] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.454420 28505 leader_election.cc:290] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f [CANDIDATE]: Term 1 election: Requested vote from peers d0fb784728cb414594ee1ee6d03e875b (127.19.84.2:38709), 6cea691cf09a44fda4b6307ef400cbd6 (127.19.84.3:36069)
I20260131 03:08:37.454598 28281 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "051b8e1ad58e497aa4950c834b54095b" candidate_uuid: "933d0320a7644b8ebac076ae68a53c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0fb784728cb414594ee1ee6d03e875b"
I20260131 03:08:37.454600 28423 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "051b8e1ad58e497aa4950c834b54095b" candidate_uuid: "933d0320a7644b8ebac076ae68a53c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cea691cf09a44fda4b6307ef400cbd6"
I20260131 03:08:37.454672 28423 raft_consensus.cc:3060] T 051b8e1ad58e497aa4950c834b54095b P 6cea691cf09a44fda4b6307ef400cbd6 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:37.454672 28281 raft_consensus.cc:3060] T 051b8e1ad58e497aa4950c834b54095b P d0fb784728cb414594ee1ee6d03e875b [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:37.455442 28423 raft_consensus.cc:2468] T 051b8e1ad58e497aa4950c834b54095b P 6cea691cf09a44fda4b6307ef400cbd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 933d0320a7644b8ebac076ae68a53c3f in term 1.
I20260131 03:08:37.455443 28281 raft_consensus.cc:2468] T 051b8e1ad58e497aa4950c834b54095b P d0fb784728cb414594ee1ee6d03e875b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 933d0320a7644b8ebac076ae68a53c3f in term 1.
I20260131 03:08:37.455646 28073 leader_election.cc:304] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 933d0320a7644b8ebac076ae68a53c3f, d0fb784728cb414594ee1ee6d03e875b; no voters:
I20260131 03:08:37.455791 28505 raft_consensus.cc:2804] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:08:37.455983 28505 raft_consensus.cc:697] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f [term 1 LEADER]: Becoming Leader. State: Replica: 933d0320a7644b8ebac076ae68a53c3f, State: Running, Role: LEADER
I20260131 03:08:37.456068 28505 consensus_queue.cc:237] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } }
I20260131 03:08:37.456766 27980 catalog_manager.cc:5654] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 933d0320a7644b8ebac076ae68a53c3f (127.19.84.1). New cstate: current_term: 1 leader_uuid: "933d0320a7644b8ebac076ae68a53c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933d0320a7644b8ebac076ae68a53c3f" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 33971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 } health_report { overall_health: UNKNOWN } } }
I20260131 03:08:37.469270 28496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:37.467335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45894 (local address 127.19.84.1:33971)
0131 03:08:37.467414 (+ 79us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:37.467416 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:37.467426 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:08:37.467533 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:37.467534 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:37.467551 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:37.467601 (+ 50us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:08:37.467809 (+ 208us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.468245 (+ 436us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:37.468778 (+ 533us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.468893 (+ 115us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:37.468924 (+ 31us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:08:37.469014 (+ 90us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:08:37.469074 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:37.469137 (+ 63us) server_negotiation.cc:300] Negotiation successful
0131 03:08:37.469157 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260131 03:08:37.473368 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:37.471499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:36069 (local address 127.19.84.1:34069)
0131 03:08:37.471647 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:37.471656 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:37.471707 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:37.471815 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:37.471815 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:37.471817 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:08:37.471986 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:37.471987 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.472573 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:37.472573 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:37.472978 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:37.472979 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.473032 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:37.473044 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:08:37.473241 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:08:37.473243 (+ 2us) client_negotiation.cc:769] Sending connection context
0131 03:08:37.473277 (+ 34us) client_negotiation.cc:240] Negotiation successful
0131 03:08:37.473308 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260131 03:08:37.473359 28493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:37.471550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:34069 (local address 127.19.84.3:36069)
0131 03:08:37.471587 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:37.471588 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:37.471705 (+ 117us) server_negotiation.cc:408] Connection header received
0131 03:08:37.471745 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:37.471746 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:37.471754 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:37.471800 (+ 46us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:08:37.472042 (+ 242us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.472514 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:37.473038 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.473143 (+ 105us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:37.473159 (+ 16us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:08:37.473179 (+ 20us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:08:37.473223 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:37.473292 (+ 69us) server_negotiation.cc:300] Negotiation successful
0131 03:08:37.473309 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260131 03:08:37.475350 28423 raft_consensus.cc:1275] T 051b8e1ad58e497aa4950c834b54095b P 6cea691cf09a44fda4b6307ef400cbd6 [term 1 FOLLOWER]: Refusing update from remote peer 933d0320a7644b8ebac076ae68a53c3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:08:37.475441 28281 raft_consensus.cc:1275] T 051b8e1ad58e497aa4950c834b54095b P d0fb784728cb414594ee1ee6d03e875b [term 1 FOLLOWER]: Refusing update from remote peer 933d0320a7644b8ebac076ae68a53c3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:08:37.475672 28576 consensus_queue.cc:1048] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0fb784728cb414594ee1ee6d03e875b" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:08:37.475785 28505 consensus_queue.cc:1048] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cea691cf09a44fda4b6307ef400cbd6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:08:37.481909 28496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:37.480075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:58507 (local address 127.19.84.1:33971)
0131 03:08:37.480127 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:37.480129 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:37.480137 (+ 8us) server_negotiation.cc:408] Connection header received
0131 03:08:37.480157 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:37.480158 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:37.480167 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:37.480248 (+ 81us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:08:37.480458 (+ 210us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.480950 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:37.481512 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.481653 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:37.481670 (+ 17us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:08:37.481687 (+ 17us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:08:37.481760 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:37.481826 (+ 66us) server_negotiation.cc:300] Negotiation successful
0131 03:08:37.481845 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260131 03:08:37.481933 28475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:37.479994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:33971 (local address 127.19.84.3:58507)
0131 03:08:37.480032 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:37.480042 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:37.480080 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:37.480251 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:37.480251 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:37.480253 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0131 03:08:37.480381 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:37.480382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.481033 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:37.481034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:37.481409 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:37.481410 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:37.481512 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:37.481516 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0131 03:08:37.481765 (+ 249us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0131 03:08:37.481767 (+ 2us) client_negotiation.cc:769] Sending connection context
0131 03:08:37.481819 (+ 52us) client_negotiation.cc:240] Negotiation successful
0131 03:08:37.481867 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260131 03:08:37.493777 27980 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:08:37.493896 27980 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:08:37.494581 27980 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 8bd0963c2d344f058b9aa4fd3fc2cce1: Sending DeleteTablet for 3 replicas of tablet 051b8e1ad58e497aa4950c834b54095b
I20260131 03:08:37.494910 28260 tablet_service.cc:1552] Processing DeleteTablet for tablet 051b8e1ad58e497aa4950c834b54095b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:37 UTC) from {username='oryx', principal='oryx/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:43784
I20260131 03:08:37.494936 28119 tablet_service.cc:1552] Processing DeleteTablet for tablet 051b8e1ad58e497aa4950c834b54095b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:37 UTC) from {username='oryx', principal='oryx/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:45880
I20260131 03:08:37.494922 28403 tablet_service.cc:1552] Processing DeleteTablet for tablet 051b8e1ad58e497aa4950c834b54095b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:08:37 UTC) from {username='oryx', principal='oryx/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:43906
I20260131 03:08:37.495010 28522 tablet_replica.cc:333] T 051b8e1ad58e497aa4950c834b54095b P d0fb784728cb414594ee1ee6d03e875b: stopping tablet replica
I20260131 03:08:37.495018 28520 tablet_replica.cc:333] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f: stopping tablet replica
I20260131 03:08:37.495025 28521 tablet_replica.cc:333] T 051b8e1ad58e497aa4950c834b54095b P 6cea691cf09a44fda4b6307ef400cbd6: stopping tablet replica
I20260131 03:08:37.495059 28520 raft_consensus.cc:2243] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:08:37.495059 28522 raft_consensus.cc:2243] T 051b8e1ad58e497aa4950c834b54095b P d0fb784728cb414594ee1ee6d03e875b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:08:37.495075 28521 raft_consensus.cc:2243] T 051b8e1ad58e497aa4950c834b54095b P 6cea691cf09a44fda4b6307ef400cbd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:08:37.495097 28522 raft_consensus.cc:2272] T 051b8e1ad58e497aa4950c834b54095b P d0fb784728cb414594ee1ee6d03e875b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:08:37.495121 28521 raft_consensus.cc:2272] T 051b8e1ad58e497aa4950c834b54095b P 6cea691cf09a44fda4b6307ef400cbd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:08:37.495148 28520 raft_consensus.cc:2272] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:08:37.495493 28522 ts_tablet_manager.cc:1916] T 051b8e1ad58e497aa4950c834b54095b P d0fb784728cb414594ee1ee6d03e875b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:08:37.495492 28520 ts_tablet_manager.cc:1916] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:08:37.495535 28521 ts_tablet_manager.cc:1916] T 051b8e1ad58e497aa4950c834b54095b P 6cea691cf09a44fda4b6307ef400cbd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:08:37.497247 28520 ts_tablet_manager.cc:1929] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:08:37.497255 28521 ts_tablet_manager.cc:1929] T 051b8e1ad58e497aa4950c834b54095b P 6cea691cf09a44fda4b6307ef400cbd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:08:37.497313 28521 log.cc:1199] T 051b8e1ad58e497aa4950c834b54095b P 6cea691cf09a44fda4b6307ef400cbd6: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/051b8e1ad58e497aa4950c834b54095b
I20260131 03:08:37.497314 28520 log.cc:1199] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-0/wal/wals/051b8e1ad58e497aa4950c834b54095b
I20260131 03:08:37.497359 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 28054
I20260131 03:08:37.497552 28520 ts_tablet_manager.cc:1950] T 051b8e1ad58e497aa4950c834b54095b P 933d0320a7644b8ebac076ae68a53c3f: Deleting consensus metadata
I20260131 03:08:37.497570 28521 ts_tablet_manager.cc:1950] T 051b8e1ad58e497aa4950c834b54095b P 6cea691cf09a44fda4b6307ef400cbd6: Deleting consensus metadata
I20260131 03:08:37.498162 27966 catalog_manager.cc:4985] TS 6cea691cf09a44fda4b6307ef400cbd6 (127.19.84.3:36069): tablet 051b8e1ad58e497aa4950c834b54095b (table test-table [id=ebe905ad7e654f89ae6f9f82e57120d9]) successfully deleted
I20260131 03:08:37.499169 28522 ts_tablet_manager.cc:1929] T 051b8e1ad58e497aa4950c834b54095b P d0fb784728cb414594ee1ee6d03e875b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260131 03:08:37.499224 28522 log.cc:1199] T 051b8e1ad58e497aa4950c834b54095b P d0fb784728cb414594ee1ee6d03e875b: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769828849314684-19792-0/minicluster-data/ts-1/wal/wals/051b8e1ad58e497aa4950c834b54095b
I20260131 03:08:37.499495 28522 ts_tablet_manager.cc:1950] T 051b8e1ad58e497aa4950c834b54095b P d0fb784728cb414594ee1ee6d03e875b: Deleting consensus metadata
I20260131 03:08:37.500012 27967 catalog_manager.cc:4985] TS d0fb784728cb414594ee1ee6d03e875b (127.19.84.2:38709): tablet 051b8e1ad58e497aa4950c834b54095b (table test-table [id=ebe905ad7e654f89ae6f9f82e57120d9]) successfully deleted
W20260131 03:08:37.503520 27966 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.84.1:33971 (error 108)
I20260131 03:08:37.503754 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 28196
I20260131 03:08:37.504941 28586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:37.504754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:33971 (local address 127.0.0.1:45910)
0131 03:08:37.504847 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:37.504888 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.1:33971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":41,"threads_started":1}
W20260131 03:08:37.505412 27966 catalog_manager.cc:4712] TS 933d0320a7644b8ebac076ae68a53c3f (127.19.84.1:33971): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 051b8e1ad58e497aa4950c834b54095b: Network error: Client connection negotiation failed: client connection to 127.19.84.1:33971: connect: Connection refused (error 111)
I20260131 03:08:37.509889 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 28338
I20260131 03:08:37.515899 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 27948
2026-01-31T03:08:37Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3740 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:08:37 dist-test-slave-5nnp krb5kdc[28589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:08:37 dist-test-slave-5nnp krb5kdc[28589](info): set up 2 sockets
Jan 31 03:08:37 dist-test-slave-5nnp krb5kdc[28589](info): commencing operation
krb5kdc: starting...
W20260131 03:08:39.532400 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.995s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:08:39 dist-test-slave-5nnp krb5kdc[28589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:08:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:08:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:08:39.675486 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:46451
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:36141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:46451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:39.754010 28605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:39.754204 28605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:39.754228 28605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:39.755764 28605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:08:39.755810 28605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:39.755825 28605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:39.755838 28605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:08:39.755851 28605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:08:39.757508 28605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:36141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:46451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:46451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:39.757761 28605 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:39.757974 28605 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:39.760637 28613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:39.760689 28611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:39.760792 28610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:39.760850 28605 server_base.cc:1060] running on GCE node
I20260131 03:08:39.761070 28605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:39.761329 28605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:39.762518 28605 hybrid_clock.cc:648] HybridClock initialized: now 1769828919762506 us; error 45 us; skew 500 ppm
W20260131 03:08:39.764319 28605 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-01-31T03:08:39Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2255 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:08:39 dist-test-slave-5nnp krb5kdc[28618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:08:39 dist-test-slave-5nnp krb5kdc[28618](info): set up 2 sockets
Jan 31 03:08:39 dist-test-slave-5nnp krb5kdc[28618](info): commencing operation
krb5kdc: starting...
W20260131 03:08:41.808559 19792 mini_kdc.cc:121] Time spent starting KDC: real 2.015s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:08:41 dist-test-slave-5nnp krb5kdc[28618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:08:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:08:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:08:41.959208 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:34775
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:34775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:42.046182 28634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:42.046401 28634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:42.046428 28634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:42.048003 28634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:08:42.048059 28634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:42.048075 28634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:42.048089 28634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:08:42.048100 28634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:08:42.049796 28634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:34775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:34775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:42.050010 28634 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:42.050161 28634 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:42.052660 28642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:42.052747 28640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:42.052742 28639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:42.052973 28634 server_base.cc:1060] running on GCE node
I20260131 03:08:42.053246 28634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:42.053508 28634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:42.054723 28634 hybrid_clock.cc:648] HybridClock initialized: now 1769828922054697 us; error 44 us; skew 500 ppm
Jan 31 03:08:42 dist-test-slave-5nnp krb5kdc[28618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828922, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:42.057459 28634 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:08:42.058073 28634 webserver.cc:492] Webserver started at http://127.19.84.62:39249/ using document root <none> and password file <none>
I20260131 03:08:42.058291 28634 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:42.058349 28634 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:42.058471 28634 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:42.059552 28634 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "c55664dcbacc4a4da7054fa8552bb6a8"
format_stamp: "Formatted at 2026-01-31 03:08:42 on dist-test-slave-5nnp"
server_key: "ef0c8947c3a8152838b41ed840e033e0"
server_key_iv: "700317f46abfce5b23ef3f83503738c7"
server_key_version: "encryptionkey@0"
I20260131 03:08:42.059903 28634 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "c55664dcbacc4a4da7054fa8552bb6a8"
format_stamp: "Formatted at 2026-01-31 03:08:42 on dist-test-slave-5nnp"
server_key: "ef0c8947c3a8152838b41ed840e033e0"
server_key_iv: "700317f46abfce5b23ef3f83503738c7"
server_key_version: "encryptionkey@0"
I20260131 03:08:42.061286 28634 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:08:42.062171 28649 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:42.062443 28634 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:42.062543 28634 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "c55664dcbacc4a4da7054fa8552bb6a8"
format_stamp: "Formatted at 2026-01-31 03:08:42 on dist-test-slave-5nnp"
server_key: "ef0c8947c3a8152838b41ed840e033e0"
server_key_iv: "700317f46abfce5b23ef3f83503738c7"
server_key_version: "encryptionkey@0"
I20260131 03:08:42.062631 28634 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:42.072435 28634 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:42.074337 28634 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:42.074522 28634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:42.078578 28634 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:34775
I20260131 03:08:42.078653 28701 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:34775 every 8 connection(s)
I20260131 03:08:42.078992 28634 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:08:42.079742 28702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:42.082310 28702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8: Bootstrap starting.
I20260131 03:08:42.082960 28702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:42.083235 28702 log.cc:826] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:42.084205 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 28634
I20260131 03:08:42.084307 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:08:42.084259 28702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8: No bootstrap required, opened a new log
I20260131 03:08:42.084504 19792 external_mini_cluster.cc:1468] Setting key c526a36de9823f02129e34f26aca19ca
I20260131 03:08:42.085891 28702 raft_consensus.cc:359] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55664dcbacc4a4da7054fa8552bb6a8" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34775 } }
I20260131 03:08:42.086053 28702 raft_consensus.cc:385] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:42.086086 28702 raft_consensus.cc:740] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c55664dcbacc4a4da7054fa8552bb6a8, State: Initialized, Role: FOLLOWER
I20260131 03:08:42.086215 28702 consensus_queue.cc:260] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55664dcbacc4a4da7054fa8552bb6a8" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34775 } }
I20260131 03:08:42.086304 28702 raft_consensus.cc:399] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:08:42.086341 28702 raft_consensus.cc:493] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:08:42.086395 28702 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:42.087198 28702 raft_consensus.cc:515] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55664dcbacc4a4da7054fa8552bb6a8" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34775 } }
I20260131 03:08:42.087301 28702 leader_election.cc:304] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c55664dcbacc4a4da7054fa8552bb6a8; no voters:
I20260131 03:08:42.087590 28702 leader_election.cc:290] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:08:42.087623 28707 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:08:42.087739 28707 raft_consensus.cc:697] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [term 1 LEADER]: Becoming Leader. State: Replica: c55664dcbacc4a4da7054fa8552bb6a8, State: Running, Role: LEADER
I20260131 03:08:42.087857 28702 sys_catalog.cc:565] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:08:42.087847 28707 consensus_queue.cc:237] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55664dcbacc4a4da7054fa8552bb6a8" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34775 } }
I20260131 03:08:42.088195 28707 sys_catalog.cc:455] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c55664dcbacc4a4da7054fa8552bb6a8. Latest consensus state: current_term: 1 leader_uuid: "c55664dcbacc4a4da7054fa8552bb6a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55664dcbacc4a4da7054fa8552bb6a8" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34775 } } }
I20260131 03:08:42.088313 28707 sys_catalog.cc:458] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:42.088326 28708 sys_catalog.cc:455] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c55664dcbacc4a4da7054fa8552bb6a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55664dcbacc4a4da7054fa8552bb6a8" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34775 } } }
I20260131 03:08:42.088402 28708 sys_catalog.cc:458] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:42.088526 28711 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:08:42.089320 28711 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:08:42.091159 28711 catalog_manager.cc:1357] Generated new cluster ID: 6a01aeb08e1542a8a95890a210ac6a5d
I20260131 03:08:42.091214 28711 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Jan 31 03:08:42 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:08:42.095942 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:42.085084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53872 (local address 127.19.84.62:34775)
0131 03:08:42.085251 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:42.085261 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:42.085282 (+ 21us) server_negotiation.cc:408] Connection header received
0131 03:08:42.085859 (+ 577us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:42.085868 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:42.085921 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:42.086041 (+ 120us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:42.087090 (+ 1049us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.088430 (+ 1340us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:42.089481 (+ 1051us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.089718 (+ 237us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:42.092224 (+ 2506us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:42.092246 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:42.092247 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:42.092269 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:42.094382 (+ 2113us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:42.094786 (+ 404us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:42.094789 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:42.094790 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:42.094865 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:42.095035 (+ 170us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:42.095036 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:42.095037 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:42.095310 (+ 273us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:42.095513 (+ 203us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:42.095605 (+ 92us) server_negotiation.cc:300] Negotiation successful
0131 03:08:42.095750 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260131 03:08:42.098412 28711 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:08:42.098951 28711 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:08:42.109766 28711 catalog_manager.cc:6027] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8: Generated new TSK 0
I20260131 03:08:42.109989 28711 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:08:42.167505 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:34775
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:42.252314 28730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:42.252490 28730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:42.252516 28730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:42.254191 28730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:42.254248 28730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:42.254278 28730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:08:42.256120 28730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:34775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:42.256387 28730 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:42.256614 28730 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:42.259338 28735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:42.259552 28738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:42.259583 28736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:42.259696 28730 server_base.cc:1060] running on GCE node
I20260131 03:08:42.259971 28730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:42.260203 28730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:42.261363 28730 hybrid_clock.cc:648] HybridClock initialized: now 1769828922261366 us; error 46 us; skew 500 ppm
Jan 31 03:08:42 dist-test-slave-5nnp krb5kdc[28618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828922, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:42.263916 28730 init.cc:377] Logged in from keytab as kudu/127.19.84.1@KRBTEST.COM (short username kudu)
I20260131 03:08:42.264482 28730 webserver.cc:492] Webserver started at http://127.19.84.1:37245/ using document root <none> and password file <none>
I20260131 03:08:42.264644 28730 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:42.264684 28730 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:42.264765 28730 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:42.265682 28730 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "0ac88858be7a434a974702a1ffaab956"
format_stamp: "Formatted at 2026-01-31 03:08:42 on dist-test-slave-5nnp"
server_key: "a2e7e557812001e81aab3ddfa1557e4b"
server_key_iv: "f7cb87d2f4742fe83cd96835290547f9"
server_key_version: "encryptionkey@0"
I20260131 03:08:42.265944 28730 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "0ac88858be7a434a974702a1ffaab956"
format_stamp: "Formatted at 2026-01-31 03:08:42 on dist-test-slave-5nnp"
server_key: "a2e7e557812001e81aab3ddfa1557e4b"
server_key_iv: "f7cb87d2f4742fe83cd96835290547f9"
server_key_version: "encryptionkey@0"
I20260131 03:08:42.267204 28730 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:08:42.268126 28745 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:42.268515 28730 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260131 03:08:42.268596 28730 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "0ac88858be7a434a974702a1ffaab956"
format_stamp: "Formatted at 2026-01-31 03:08:42 on dist-test-slave-5nnp"
server_key: "a2e7e557812001e81aab3ddfa1557e4b"
server_key_iv: "f7cb87d2f4742fe83cd96835290547f9"
server_key_version: "encryptionkey@0"
I20260131 03:08:42.268704 28730 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:42.277839 28730 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:42.279994 28730 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:42.280133 28730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:42.280395 28730 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:42.280743 28730 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:42.280777 28730 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:42.280802 28730 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:42.280850 28730 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:42.287132 28730 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:34539
I20260131 03:08:42.287168 28858 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:34539 every 8 connection(s)
I20260131 03:08:42.287655 28730 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
I20260131 03:08:42.292207 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 28730
I20260131 03:08:42.292347 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:08:42.292559 19792 external_mini_cluster.cc:1468] Setting key 88cdcf7dab0a2bc2308117f58b7f5461
Jan 31 03:08:42 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828922, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:08:42.297186 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:42.288442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:58827 (local address 127.19.84.62:34775)
0131 03:08:42.288522 (+ 80us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:42.288525 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:42.288673 (+ 148us) server_negotiation.cc:408] Connection header received
0131 03:08:42.289208 (+ 535us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:42.289209 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:42.289223 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:42.289296 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:42.290061 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.290583 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:42.291130 (+ 547us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.291244 (+ 114us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:42.293775 (+ 2531us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:42.293780 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:42.293781 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:42.293788 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:42.295524 (+ 1736us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:42.296037 (+ 513us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:42.296038 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:42.296038 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:42.296063 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:42.296324 (+ 261us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:42.296325 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:42.296326 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:42.296483 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:42.296552 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:42.296949 (+ 397us) server_negotiation.cc:300] Negotiation successful
0131 03:08:42.297040 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260131 03:08:42.297289 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:42.288391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.1:58827)
0131 03:08:42.288546 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:42.288581 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:42.289104 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:42.289375 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:42.289382 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:42.289710 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:42.289979 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:42.289982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.290664 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:42.290664 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:42.291062 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:42.291063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.291433 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:42.292024 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:42.292043 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:42.293695 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:42.295611 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:42.295618 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:42.295621 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:42.295938 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:42.296141 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:42.296142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:42.296143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:42.296236 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:42.296569 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:42.296575 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:42.296757 (+ 182us) client_negotiation.cc:769] Sending connection context
0131 03:08:42.296902 (+ 145us) client_negotiation.cc:240] Negotiation successful
0131 03:08:42.297068 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
I20260131 03:08:42.297891 28859 heartbeater.cc:344] Connected to a master server at 127.19.84.62:34775
I20260131 03:08:42.298023 28859 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:42.298246 28859 heartbeater.cc:507] Master 127.19.84.62:34775 requested a full tablet report, sending...
I20260131 03:08:42.298813 28666 ts_manager.cc:194] Registered new tserver with Master: 0ac88858be7a434a974702a1ffaab956 (127.19.84.1:34539)
I20260131 03:08:42.299559 28666 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.1@KRBTEST.COM'} at 127.19.84.1:58827
WARNING: no policy specified for kudu/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:08:42.342474 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:34775
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:42.422463 28866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:42.422668 28866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:42.422719 28866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:42.424173 28866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:42.424252 28866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:42.424297 28866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:08:42.425911 28866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:34775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:42.426170 28866 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:42.426390 28866 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:42.429153 28871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:42.429129 28872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:42.429143 28866 server_base.cc:1060] running on GCE node
W20260131 03:08:42.429131 28874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:42.429518 28866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:42.429690 28866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:42.430889 28866 hybrid_clock.cc:648] HybridClock initialized: now 1769828922430878 us; error 33 us; skew 500 ppm
Jan 31 03:08:42 dist-test-slave-5nnp krb5kdc[28618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828922, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:42.433424 28866 init.cc:377] Logged in from keytab as kudu/127.19.84.2@KRBTEST.COM (short username kudu)
I20260131 03:08:42.433979 28866 webserver.cc:492] Webserver started at http://127.19.84.2:38127/ using document root <none> and password file <none>
I20260131 03:08:42.434134 28866 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:42.434163 28866 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:42.434240 28866 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:42.435137 28866 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "d0b92451a7654cab8c604e6882bfd0ef"
format_stamp: "Formatted at 2026-01-31 03:08:42 on dist-test-slave-5nnp"
server_key: "eb737e55a4e877cf7cfc6e377b0684bc"
server_key_iv: "0b94481ff95e17e998e4aee395cbe0c2"
server_key_version: "encryptionkey@0"
I20260131 03:08:42.435469 28866 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "d0b92451a7654cab8c604e6882bfd0ef"
format_stamp: "Formatted at 2026-01-31 03:08:42 on dist-test-slave-5nnp"
server_key: "eb737e55a4e877cf7cfc6e377b0684bc"
server_key_iv: "0b94481ff95e17e998e4aee395cbe0c2"
server_key_version: "encryptionkey@0"
I20260131 03:08:42.436684 28866 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:42.437345 28881 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:42.437526 28866 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260131 03:08:42.437618 28866 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "d0b92451a7654cab8c604e6882bfd0ef"
format_stamp: "Formatted at 2026-01-31 03:08:42 on dist-test-slave-5nnp"
server_key: "eb737e55a4e877cf7cfc6e377b0684bc"
server_key_iv: "0b94481ff95e17e998e4aee395cbe0c2"
server_key_version: "encryptionkey@0"
I20260131 03:08:42.437716 28866 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:42.444650 28866 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:42.446420 28866 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:42.446645 28866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:42.446887 28866 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:42.447227 28866 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:42.447284 28866 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:42.447335 28866 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:42.447358 28866 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:42.454200 28866 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:39329
I20260131 03:08:42.454236 28994 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:39329 every 8 connection(s)
I20260131 03:08:42.454600 28866 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
I20260131 03:08:42.456974 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 28866
I20260131 03:08:42.457082 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:08:42.457261 19792 external_mini_cluster.cc:1468] Setting key c159547f8ec25de556d6441d512cae96
Jan 31 03:08:42 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828922, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:08:42.464104 28997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:42.455320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.2:56365)
0131 03:08:42.455500 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:42.455530 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:42.455992 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:42.456219 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:42.456223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:42.456553 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:42.456845 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:42.456847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.457817 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:42.457818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:42.458334 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:42.458336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.458777 (+ 441us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:42.459366 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:42.459439 (+ 73us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:42.460906 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:42.462778 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:42.462783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:42.462785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:42.462991 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:42.463215 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:42.463216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:42.463217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:42.463289 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:42.463594 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:42.463600 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:42.463741 (+ 141us) client_negotiation.cc:769] Sending connection context
0131 03:08:42.463843 (+ 102us) client_negotiation.cc:240] Negotiation successful
0131 03:08:42.463956 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":128,"threads_started":1}
I20260131 03:08:42.464114 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:42.455369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:56365 (local address 127.19.84.62:34775)
0131 03:08:42.455463 (+ 94us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:42.455466 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:42.455622 (+ 156us) server_negotiation.cc:408] Connection header received
0131 03:08:42.456099 (+ 477us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:42.456100 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:42.456111 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:42.456175 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:42.456959 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.457713 (+ 754us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:42.458426 (+ 713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.458663 (+ 237us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:42.461003 (+ 2340us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:42.461008 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:42.461008 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:42.461018 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:42.462677 (+ 1659us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:42.463088 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:42.463089 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:42.463089 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:42.463119 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:42.463368 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:42.463369 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:42.463370 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:42.463515 (+ 145us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:42.463597 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:42.463884 (+ 287us) server_negotiation.cc:300] Negotiation successful
0131 03:08:42.464011 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260131 03:08:42.464829 28995 heartbeater.cc:344] Connected to a master server at 127.19.84.62:34775
I20260131 03:08:42.464946 28995 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:42.465196 28995 heartbeater.cc:507] Master 127.19.84.62:34775 requested a full tablet report, sending...
I20260131 03:08:42.465744 28666 ts_manager.cc:194] Registered new tserver with Master: d0b92451a7654cab8c604e6882bfd0ef (127.19.84.2:39329)
WARNING: no policy specified for kudu/127.19.84.3@KRBTEST.COM; defaulting to no policy
I20260131 03:08:42.466236 28666 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.2@KRBTEST.COM'} at 127.19.84.2:56365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:08:42.507261 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:34775
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:42.587121 29002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:42.587355 29002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:42.587417 29002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:42.588907 29002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:42.588989 29002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:42.589033 29002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:08:42.590804 29002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:34775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:42.591070 29002 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:42.591281 29002 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:42.594070 29010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:42.594197 29002 server_base.cc:1060] running on GCE node
W20260131 03:08:42.594081 29007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:42.594081 29008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:42.594516 29002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:42.594709 29002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:42.595880 29002 hybrid_clock.cc:648] HybridClock initialized: now 1769828922595866 us; error 30 us; skew 500 ppm
Jan 31 03:08:42 dist-test-slave-5nnp krb5kdc[28618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828922, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:42.598423 29002 init.cc:377] Logged in from keytab as kudu/127.19.84.3@KRBTEST.COM (short username kudu)
I20260131 03:08:42.599025 29002 webserver.cc:492] Webserver started at http://127.19.84.3:39169/ using document root <none> and password file <none>
I20260131 03:08:42.599223 29002 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:42.599279 29002 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:42.599452 29002 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:42.600428 29002 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "a74c2b304ff74aefb956f127650b2cff"
format_stamp: "Formatted at 2026-01-31 03:08:42 on dist-test-slave-5nnp"
server_key: "58ce74d2c1e6caf6c3beb6fb01263ff9"
server_key_iv: "97b332e1dda84cb2a58a23a2f7e5def1"
server_key_version: "encryptionkey@0"
I20260131 03:08:42.600769 29002 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "a74c2b304ff74aefb956f127650b2cff"
format_stamp: "Formatted at 2026-01-31 03:08:42 on dist-test-slave-5nnp"
server_key: "58ce74d2c1e6caf6c3beb6fb01263ff9"
server_key_iv: "97b332e1dda84cb2a58a23a2f7e5def1"
server_key_version: "encryptionkey@0"
I20260131 03:08:42.602022 29002 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:08:42.602767 29017 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:42.602986 29002 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:42.603073 29002 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "a74c2b304ff74aefb956f127650b2cff"
format_stamp: "Formatted at 2026-01-31 03:08:42 on dist-test-slave-5nnp"
server_key: "58ce74d2c1e6caf6c3beb6fb01263ff9"
server_key_iv: "97b332e1dda84cb2a58a23a2f7e5def1"
server_key_version: "encryptionkey@0"
I20260131 03:08:42.603154 29002 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:42.612607 29002 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:42.614192 29002 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:42.614346 29002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:42.614562 29002 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:42.614905 29002 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:42.614962 29002 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:42.615002 29002 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:42.615034 29002 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:42.620867 29002 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:35655
I20260131 03:08:42.620908 29130 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:35655 every 8 connection(s)
I20260131 03:08:42.621199 29002 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
I20260131 03:08:42.621999 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 29002
I20260131 03:08:42.622100 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:08:42.622288 19792 external_mini_cluster.cc:1468] Setting key 72e45ef8ebcce0dce9949cd12b0c15d3
Jan 31 03:08:42 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828922, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:08:42.631148 28706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:42.621973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:50775 (local address 127.19.84.62:34775)
0131 03:08:42.622047 (+ 74us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:42.622049 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:42.622229 (+ 180us) server_negotiation.cc:408] Connection header received
0131 03:08:42.622978 (+ 749us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:42.622979 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:42.622993 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:42.623081 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:42.624256 (+ 1175us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.625194 (+ 938us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:42.626091 (+ 897us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.626222 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:42.628442 (+ 2220us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:42.628450 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:42.628451 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:42.628458 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:42.629826 (+ 1368us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:42.630269 (+ 443us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:42.630270 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:42.630271 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:42.630303 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:42.630506 (+ 203us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:42.630507 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:42.630507 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:42.630610 (+ 103us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:42.630685 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:42.630959 (+ 274us) server_negotiation.cc:300] Negotiation successful
0131 03:08:42.631055 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260131 03:08:42.631278 29133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:42.621934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.3:50775)
0131 03:08:42.622110 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:42.622143 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:42.622818 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:42.623146 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:42.623150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:42.623651 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:42.624122 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:42.624126 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.625380 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:42.625381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:42.626015 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:42.626017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.626369 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:42.626853 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:42.626870 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:42.628324 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:42.629954 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:42.629961 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:42.629964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:42.630180 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:42.630368 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:08:42.630369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:08:42.630370 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:08:42.630442 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:08:42.630688 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:42.630694 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:42.630828 (+ 134us) client_negotiation.cc:769] Sending connection context
0131 03:08:42.630942 (+ 114us) client_negotiation.cc:240] Negotiation successful
0131 03:08:42.631080 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":73,"threads_started":1}
I20260131 03:08:42.631861 29131 heartbeater.cc:344] Connected to a master server at 127.19.84.62:34775
I20260131 03:08:42.631944 29131 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:42.632107 29131 heartbeater.cc:507] Master 127.19.84.62:34775 requested a full tablet report, sending...
I20260131 03:08:42.632424 28666 ts_manager.cc:194] Registered new tserver with Master: a74c2b304ff74aefb956f127650b2cff (127.19.84.3:35655)
I20260131 03:08:42.632761 28666 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:50775
I20260131 03:08:42.635741 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/oryx_127.19.84.1.keytab.
Entry for principal oryx/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/oryx_127.19.84.1.keytab.
WARNING: no policy specified for oryx/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/oryx_127.19.84.2.keytab.
Entry for principal oryx/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/oryx_127.19.84.2.keytab.
WARNING: no policy specified for oryx/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/oryx_127.19.84.3.keytab.
Entry for principal oryx/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/oryx_127.19.84.3.keytab.
I20260131 03:08:42.705606 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 28730
I20260131 03:08:42.710073 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 28866
I20260131 03:08:42.714313 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 29002
I20260131 03:08:42.718755 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 28634
I20260131 03:08:42.724112 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:34775
--webserver_interface=127.19.84.62
--webserver_port=39249
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:34775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:08:42.803998 29140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:42.804205 29140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:42.804231 29140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:42.805763 29140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:08:42.805816 29140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:42.805831 29140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:42.805845 29140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:08:42.805857 29140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:08:42.807576 29140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:34775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:34775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=39249
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:42.807835 29140 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:42.808017 29140 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:42.810712 29148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:42.810803 29146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:42.810825 29140 server_base.cc:1060] running on GCE node
W20260131 03:08:42.810950 29145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:42.811178 29140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:42.811456 29140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:42.812640 29140 hybrid_clock.cc:648] HybridClock initialized: now 1769828922812616 us; error 41 us; skew 500 ppm
Jan 31 03:08:42 dist-test-slave-5nnp krb5kdc[28618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828922, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:42.815279 29140 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:08:42.815923 29140 webserver.cc:492] Webserver started at http://127.19.84.62:39249/ using document root <none> and password file <none>
I20260131 03:08:42.816144 29140 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:42.816193 29140 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:42.817636 29140 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:42.818449 29155 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:42.818688 29140 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:42.818795 29140 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "c55664dcbacc4a4da7054fa8552bb6a8"
format_stamp: "Formatted at 2026-01-31 03:08:42 on dist-test-slave-5nnp"
server_key: "ef0c8947c3a8152838b41ed840e033e0"
server_key_iv: "700317f46abfce5b23ef3f83503738c7"
server_key_version: "encryptionkey@0"
I20260131 03:08:42.819123 29140 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:42.826876 29140 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:42.829241 29140 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:42.829452 29140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:42.833726 29140 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:34775
I20260131 03:08:42.833751 29207 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:34775 every 8 connection(s)
I20260131 03:08:42.834108 29140 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:08:42.835312 29208 sys_catalog.cc:263] Verifying existing consensus state
I20260131 03:08:42.835952 29208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8: Bootstrap starting.
I20260131 03:08:42.838364 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 29140
I20260131 03:08:42.838663 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:34539
--local_ip_for_outbound_sockets=127.19.84.1
--tserver_master_addrs=127.19.84.62:34775
--webserver_port=37245
--webserver_interface=127.19.84.1
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.84.1
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/oryx_127.19.84.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260131 03:08:42.844928 29208 log.cc:826] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:42.846618 29208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260131 03:08:42.846933 29208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8: Bootstrap complete.
I20260131 03:08:42.849102 29208 raft_consensus.cc:359] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55664dcbacc4a4da7054fa8552bb6a8" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34775 } }
I20260131 03:08:42.849296 29208 raft_consensus.cc:740] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c55664dcbacc4a4da7054fa8552bb6a8, State: Initialized, Role: FOLLOWER
I20260131 03:08:42.849491 29208 consensus_queue.cc:260] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55664dcbacc4a4da7054fa8552bb6a8" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34775 } }
I20260131 03:08:42.849562 29208 raft_consensus.cc:399] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:08:42.849623 29208 raft_consensus.cc:493] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:08:42.849673 29208 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [term 1 FOLLOWER]: Advancing to term 2
I20260131 03:08:42.850548 29208 raft_consensus.cc:515] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55664dcbacc4a4da7054fa8552bb6a8" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34775 } }
I20260131 03:08:42.850634 29208 leader_election.cc:304] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c55664dcbacc4a4da7054fa8552bb6a8; no voters:
I20260131 03:08:42.850863 29208 leader_election.cc:290] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260131 03:08:42.850903 29212 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [term 2 FOLLOWER]: Leader election won for term 2
I20260131 03:08:42.851007 29212 raft_consensus.cc:697] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [term 2 LEADER]: Becoming Leader. State: Replica: c55664dcbacc4a4da7054fa8552bb6a8, State: Running, Role: LEADER
I20260131 03:08:42.851087 29212 consensus_queue.cc:237] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55664dcbacc4a4da7054fa8552bb6a8" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34775 } }
I20260131 03:08:42.851111 29208 sys_catalog.cc:565] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:08:42.851640 29212 sys_catalog.cc:455] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c55664dcbacc4a4da7054fa8552bb6a8. Latest consensus state: current_term: 2 leader_uuid: "c55664dcbacc4a4da7054fa8552bb6a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55664dcbacc4a4da7054fa8552bb6a8" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34775 } } }
I20260131 03:08:42.851804 29212 sys_catalog.cc:458] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:42.851686 29214 sys_catalog.cc:455] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c55664dcbacc4a4da7054fa8552bb6a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55664dcbacc4a4da7054fa8552bb6a8" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34775 } } }
I20260131 03:08:42.851959 29214 sys_catalog.cc:458] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:42.852110 29221 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:08:42.852758 29221 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:08:42.852881 29221 catalog_manager.cc:1269] Loaded cluster ID: 6a01aeb08e1542a8a95890a210ac6a5d
I20260131 03:08:42.852913 29221 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:08:42.854238 29221 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:08:42.854445 29221 catalog_manager.cc:6038] T 00000000000000000000000000000000 P c55664dcbacc4a4da7054fa8552bb6a8: Loaded TSK: 0
I20260131 03:08:42.854637 29221 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260131 03:08:42.919715 29210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:42.919898 29210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:42.919924 29210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:42.921489 29210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:42.921546 29210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:42.921576 29210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:08:42.923473 29210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/oryx_127.19.84.1.keytab
--principal=oryx/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:34539
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=37245
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:34775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:42.923779 29210 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:42.924002 29210 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260131 03:08:42.927510 29210 server_base.cc:1060] running on GCE node
W20260131 03:08:42.927493 29234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:42.927495 29235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:42.927493 29237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:42.927994 29210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:42.928344 29210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:42.929563 29210 hybrid_clock.cc:648] HybridClock initialized: now 1769828922929547 us; error 37 us; skew 500 ppm
Jan 31 03:08:42 dist-test-slave-5nnp krb5kdc[28618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828922, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:42.932698 29210 init.cc:377] Logged in from keytab as oryx/127.19.84.1@KRBTEST.COM (short username oryx)
I20260131 03:08:42.933511 29210 webserver.cc:492] Webserver started at http://127.19.84.1:37245/ using document root <none> and password file <none>
I20260131 03:08:42.933748 29210 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:42.933822 29210 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:42.935547 29210 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:08:42.936280 29244 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:42.936592 29210 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:42.936679 29210 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "0ac88858be7a434a974702a1ffaab956"
format_stamp: "Formatted at 2026-01-31 03:08:42 on dist-test-slave-5nnp"
server_key: "a2e7e557812001e81aab3ddfa1557e4b"
server_key_iv: "f7cb87d2f4742fe83cd96835290547f9"
server_key_version: "encryptionkey@0"
I20260131 03:08:42.937021 29210 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:42.952154 29210 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:42.954748 29210 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:42.954909 29210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:42.955219 29210 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:42.955665 29210 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:42.955726 29210 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:42.955761 29210 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:42.955787 29210 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:42.963173 29210 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:34539
I20260131 03:08:42.963330 29357 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:34539 every 8 connection(s)
I20260131 03:08:42.963649 29210 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
Jan 31 03:08:42 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:42 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:42.972158 29360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:42.964439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.1:42403)
0131 03:08:42.964629 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:42.964667 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:42.965282 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:42.965689 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:42.965693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:42.966132 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:42.966538 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:42.966542 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.967634 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:42.967635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:42.968235 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:42.968238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.968397 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:42.969494 (+ 1097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:42.969517 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:42.971982 (+ 2465us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":80,"threads_started":1}
W20260131 03:08:42.972360 29358 heartbeater.cc:646] Failed to heartbeat to 127.19.84.62:34775 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.84.62:34775: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
W20260131 03:08:42.972257 29361 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:42.964474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:42403 (local address 127.19.84.62:34775)
0131 03:08:42.964652 (+ 178us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:42.964667 (+ 15us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:42.964749 (+ 82us) server_negotiation.cc:408] Connection header received
0131 03:08:42.965472 (+ 723us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:42.965477 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:42.965515 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:42.965615 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:42.966685 (+ 1070us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.967536 (+ 851us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:42.968341 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.968527 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:42.972108 (+ 3581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.1:42403: BlockingRecv error: recv got EOF from 127.19.84.1:42403 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260131 03:08:42.973577 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 29210
I20260131 03:08:42.973873 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:39329
--local_ip_for_outbound_sockets=127.19.84.2
--tserver_master_addrs=127.19.84.62:34775
--webserver_port=38127
--webserver_interface=127.19.84.2
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.84.2
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/oryx_127.19.84.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jan 31 03:08:42 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:42 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:42.977376 29360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:42.972611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.1:40311)
0131 03:08:42.972721 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:42.972733 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:42.972787 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:42.972940 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:42.972940 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:42.973069 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:42.973212 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:42.973213 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.974137 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:42.974138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:42.974724 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:42.974726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.974799 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:42.975248 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:42.975254 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:42.977283 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94}
W20260131 03:08:42.977473 29361 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:42.972689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:40311 (local address 127.19.84.62:34775)
0131 03:08:42.972725 (+ 36us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:42.972728 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:42.972785 (+ 57us) server_negotiation.cc:408] Connection header received
0131 03:08:42.972850 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:42.972851 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:42.972860 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:42.972936 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:42.973291 (+ 355us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.974046 (+ 755us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:42.974822 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.975004 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:42.977413 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.1:40311: BlockingRecv error: recv got EOF from 127.19.84.1:40311 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Jan 31 03:08:42 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:42 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:42.983067 29360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:42.977663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.1:47089)
0131 03:08:42.977817 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:42.977828 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:42.977879 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:42.978028 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:42.978029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:42.978189 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:42.978374 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:42.978376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.979288 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:42.979289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:42.979937 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:42.979939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.980022 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:42.980616 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:42.980622 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:42.982965 (+ 2343us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141}
W20260131 03:08:42.983162 29361 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:42.977753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:47089 (local address 127.19.84.62:34775)
0131 03:08:42.977791 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:42.977793 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:42.977890 (+ 97us) server_negotiation.cc:408] Connection header received
0131 03:08:42.977925 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:42.977926 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:42.977939 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:42.978008 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:42.978478 (+ 470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.979192 (+ 714us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:42.980029 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:42.980199 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:42.983092 (+ 2893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.1:47089: BlockingRecv error: recv got EOF from 127.19.84.1:47089 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260131 03:08:42.983208 29358 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260131 03:08:43.059002 29362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:43.059183 29362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:43.059211 29362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:43.060773 29362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:43.060835 29362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:43.060865 29362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:08:43.062491 29362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/oryx_127.19.84.2.keytab
--principal=oryx/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:39329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=38127
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:34775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:43.062693 29362 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:43.062844 29362 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:43.065588 29370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:43.065605 29362 server_base.cc:1060] running on GCE node
W20260131 03:08:43.065591 29368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:43.065588 29367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:43.066063 29362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:43.066255 29362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:43.067466 29362 hybrid_clock.cc:648] HybridClock initialized: now 1769828923067442 us; error 41 us; skew 500 ppm
Jan 31 03:08:43 dist-test-slave-5nnp krb5kdc[28618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828923, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:43.070142 29362 init.cc:377] Logged in from keytab as oryx/127.19.84.2@KRBTEST.COM (short username oryx)
I20260131 03:08:43.070842 29362 webserver.cc:492] Webserver started at http://127.19.84.2:38127/ using document root <none> and password file <none>
I20260131 03:08:43.071105 29362 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:43.071185 29362 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:43.072629 29362 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:43.073240 29377 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:43.073432 29362 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260131 03:08:43.073527 29362 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "d0b92451a7654cab8c604e6882bfd0ef"
format_stamp: "Formatted at 2026-01-31 03:08:42 on dist-test-slave-5nnp"
server_key: "eb737e55a4e877cf7cfc6e377b0684bc"
server_key_iv: "0b94481ff95e17e998e4aee395cbe0c2"
server_key_version: "encryptionkey@0"
I20260131 03:08:43.073846 29362 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:43.085855 29362 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:43.088025 29362 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:43.088188 29362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:43.088460 29362 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:43.088773 29362 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:43.088919 29362 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:43.089007 29362 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:43.089036 29362 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:43.095327 29362 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:39329
I20260131 03:08:43.095388 29490 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:39329 every 8 connection(s)
I20260131 03:08:43.095784 29362 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
I20260131 03:08:43.099414 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 29362
I20260131 03:08:43.099776 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:35655
--local_ip_for_outbound_sockets=127.19.84.3
--tserver_master_addrs=127.19.84.62:34775
--webserver_port=39169
--webserver_interface=127.19.84.3
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.84.3
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/oryx_127.19.84.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jan 31 03:08:43 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:43 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:43.103729 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:43.096760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.2:60253)
0131 03:08:43.096978 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:43.097014 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:43.097489 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:43.097764 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:43.097768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:43.098089 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:43.098370 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:43.098373 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.099301 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:43.099302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:43.099980 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:43.099983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.100104 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:43.101077 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:43.101099 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:43.103575 (+ 2476us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":95,"threads_started":1}
W20260131 03:08:43.103807 29361 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:43.096853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:60253 (local address 127.19.84.62:34775)
0131 03:08:43.096945 (+ 92us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:43.096948 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:43.097130 (+ 182us) server_negotiation.cc:408] Connection header received
0131 03:08:43.097596 (+ 466us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:43.097597 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:43.097613 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:43.097682 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:43.098495 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.099190 (+ 695us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:43.100066 (+ 876us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.100274 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:43.103719 (+ 3445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.2:60253: BlockingRecv error: recv got EOF from 127.19.84.2:60253 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260131 03:08:43.104219 29491 heartbeater.cc:646] Failed to heartbeat to 127.19.84.62:34775 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.84.62:34775: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Jan 31 03:08:43 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:43 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:43.109359 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:43.104490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.2:57871)
0131 03:08:43.104552 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:43.104565 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:43.104608 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:43.104815 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:43.104817 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:43.105002 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:43.105240 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:43.105243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.105941 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:43.105943 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:43.106622 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:43.106625 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.106708 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:43.107183 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:43.107189 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:43.109260 (+ 2071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260131 03:08:43.109412 29361 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:43.104619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:57871 (local address 127.19.84.62:34775)
0131 03:08:43.104684 (+ 65us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:43.104686 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:43.104695 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:08:43.104719 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:43.104720 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:43.104730 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:43.104786 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:43.105364 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.105847 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:43.106712 (+ 865us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.106825 (+ 113us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:43.109350 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.2:57871: BlockingRecv error: recv got EOF from 127.19.84.2:57871 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jan 31 03:08:43 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:43 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:43.114392 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:43.109793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.2:41841)
0131 03:08:43.109921 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:43.109933 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:43.109992 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:43.110155 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:43.110156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:43.110314 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:43.110505 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:43.110507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.111161 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:43.111162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:43.111760 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:43.111763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.111844 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:43.112345 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:43.112350 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:43.114318 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109}
W20260131 03:08:43.114483 29361 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:43.109868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:41841 (local address 127.19.84.62:34775)
0131 03:08:43.109924 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:43.109926 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:43.109987 (+ 61us) server_negotiation.cc:408] Connection header received
0131 03:08:43.110071 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:43.110071 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:43.110080 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:43.110125 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:43.110613 (+ 488us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.111073 (+ 460us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:43.111855 (+ 782us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.112021 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:43.114410 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.2:41841: BlockingRecv error: recv got EOF from 127.19.84.2:41841 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260131 03:08:43.114686 29491 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260131 03:08:43.182813 29494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:43.183033 29494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:43.183072 29494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:43.184598 29494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:43.184677 29494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:43.184721 29494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:08:43.186352 29494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/krb5kdc/oryx_127.19.84.3.keytab
--principal=oryx/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:35655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=39169
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:34775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:43.186637 29494 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:43.186843 29494 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:43.189646 29502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:43.189646 29499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:43.189729 29500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:43.189882 29494 server_base.cc:1060] running on GCE node
I20260131 03:08:43.190038 29494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:43.190227 29494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:43.191437 29494 hybrid_clock.cc:648] HybridClock initialized: now 1769828923191418 us; error 30 us; skew 500 ppm
Jan 31 03:08:43 dist-test-slave-5nnp krb5kdc[28618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828923, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:08:43.193883 29494 init.cc:377] Logged in from keytab as oryx/127.19.84.3@KRBTEST.COM (short username oryx)
I20260131 03:08:43.194447 29494 webserver.cc:492] Webserver started at http://127.19.84.3:39169/ using document root <none> and password file <none>
I20260131 03:08:43.194650 29494 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:43.194720 29494 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:43.196251 29494 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260131 03:08:43.197352 29509 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:43.197583 29494 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:43.197681 29494 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "a74c2b304ff74aefb956f127650b2cff"
format_stamp: "Formatted at 2026-01-31 03:08:42 on dist-test-slave-5nnp"
server_key: "58ce74d2c1e6caf6c3beb6fb01263ff9"
server_key_iv: "97b332e1dda84cb2a58a23a2f7e5def1"
server_key_version: "encryptionkey@0"
I20260131 03:08:43.197970 29494 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:43.208801 29494 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:43.210675 29494 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:43.210832 29494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:43.211066 29494 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:43.211453 29494 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:43.211509 29494 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:43.211553 29494 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:43.211581 29494 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:43.217875 29494 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:35655
I20260131 03:08:43.217924 29622 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:35655 every 8 connection(s)
I20260131 03:08:43.218281 29494 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
Jan 31 03:08:43 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:43 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:43.224812 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 29494
W20260131 03:08:43.225555 29361 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:43.219051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:53107 (local address 127.19.84.62:34775)
0131 03:08:43.219177 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:43.219179 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:43.219319 (+ 140us) server_negotiation.cc:408] Connection header received
0131 03:08:43.219911 (+ 592us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:43.219913 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:43.219925 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:43.219976 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:43.220739 (+ 763us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.221224 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:43.221819 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.221985 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:43.225503 (+ 3518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.3:53107: BlockingRecv error: recv got EOF from 127.19.84.3:53107 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260131 03:08:43.225596 29625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:43.219012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.3:53107)
0131 03:08:43.219193 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:43.219233 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:43.219799 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:43.220040 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:43.220043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:43.220353 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:43.220629 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:43.220632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.221319 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:43.221320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:43.221730 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:43.221732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.221886 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:43.222950 (+ 1064us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:43.222968 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:43.225426 (+ 2458us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":75,"threads_started":1}
W20260131 03:08:43.225802 29623 heartbeater.cc:646] Failed to heartbeat to 127.19.84.62:34775 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.84.62:34775: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Jan 31 03:08:43 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:43 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:43.230404 29625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:43.226044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.3:49121)
0131 03:08:43.226089 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:43.226098 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:43.226144 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:43.226285 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:43.226286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:43.226398 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:43.226555 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:43.226556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.227313 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:43.227314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:43.227856 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:43.227858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.227912 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:43.228344 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:43.228348 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:43.230326 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260131 03:08:43.230506 29361 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:43.226097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:49121 (local address 127.19.84.62:34775)
0131 03:08:43.226154 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:43.226156 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:43.226165 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:08:43.226195 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:43.226196 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:43.226205 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:43.226257 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:43.226643 (+ 386us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.227229 (+ 586us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:43.227926 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.228082 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:43.230441 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.3:49121: BlockingRecv error: recv got EOF from 127.19.84.3:49121 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260131 03:08:43.233147 29627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:43.225385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53888 (local address 127.19.84.62:34775)
0131 03:08:43.225515 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:43.225518 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:43.225910 (+ 392us) server_negotiation.cc:408] Connection header received
0131 03:08:43.225938 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:43.225939 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:43.225951 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:43.226026 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:43.226927 (+ 901us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.227586 (+ 659us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:43.228268 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.228481 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:43.229583 (+ 1102us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:43.229590 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:43.229590 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:08:43.229629 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:43.231760 (+ 2131us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:43.232055 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:43.232058 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:43.232059 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:43.232132 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:08:43.232326 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:08:43.232327 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:08:43.232327 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:08:43.232511 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:43.232664 (+ 153us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:43.232924 (+ 260us) server_negotiation.cc:300] Negotiation successful
0131 03:08:43.233052 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":83,"threads_started":1}
Jan 31 03:08:43 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:43 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:43.234784 29625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:43.230852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.3:47033)
0131 03:08:43.230908 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:43.230920 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:43.230961 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:43.231105 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:43.231106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:43.231209 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:43.231317 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:43.231318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.232147 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:43.232148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:43.232493 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:43.232494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.232548 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:43.232925 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:43.232929 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:43.234705 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260131 03:08:43.234831 29361 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:43.230925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:47033 (local address 127.19.84.62:34775)
0131 03:08:43.230990 (+ 65us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:43.230992 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:43.231001 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:08:43.231020 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:43.231020 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:43.231029 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:43.231087 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:43.231410 (+ 323us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.232078 (+ 668us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:43.232566 (+ 488us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.232724 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:43.234772 (+ 2048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.3:47033: BlockingRecv error: recv got EOF from 127.19.84.3:47033 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260131 03:08:43.234989 29623 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jan 31 03:08:43 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:43 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:43.989110 29628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:43.983765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.1:44841)
0131 03:08:43.983909 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:43.983924 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:43.984011 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:43.984267 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:43.984267 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:43.984607 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:43.984825 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:43.984827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.985794 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:43.985795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:43.986440 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:43.986443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.986506 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:43.987050 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:43.987062 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:43.989042 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
W20260131 03:08:43.989303 29629 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:43.983876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:44841 (local address 127.19.84.62:34775)
0131 03:08:43.984012 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:43.984015 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:43.984029 (+ 14us) server_negotiation.cc:408] Connection header received
0131 03:08:43.984116 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:43.984117 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:43.984149 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:43.984228 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:43.984969 (+ 741us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.985684 (+ 715us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:43.986536 (+ 852us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:43.986687 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:43.989222 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.1:44841: BlockingRecv error: recv got EOF from 127.19.84.1:44841 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
Jan 31 03:08:44 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:44 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:44.119948 29630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:44.115116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.2:38345)
0131 03:08:44.115303 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:44.115317 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:44.115464 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:44.115662 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:44.115663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:44.115848 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:44.116035 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:44.116038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:44.116741 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:44.116743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:44.117140 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:44.117142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:44.117229 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:44.117879 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:44.117885 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:44.119845 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":88,"threads_started":1}
W20260131 03:08:44.119990 29629 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:44.115222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:38345 (local address 127.19.84.62:34775)
0131 03:08:44.115309 (+ 87us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:44.115312 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:44.115435 (+ 123us) server_negotiation.cc:408] Connection header received
0131 03:08:44.115553 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:44.115554 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:44.115574 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:44.115638 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:44.116135 (+ 497us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:44.116652 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:44.117226 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:44.117440 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:44.119927 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.2:38345: BlockingRecv error: recv got EOF from 127.19.84.2:38345 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jan 31 03:08:44 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:44 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:44.239818 29631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:44.235517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.3:36583)
0131 03:08:44.235654 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:44.235672 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:44.235728 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:44.235862 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:44.235863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:44.236031 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:44.236215 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:44.236217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:44.236889 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:44.236890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:44.237388 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:44.237391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:44.237456 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:44.237899 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:44.237903 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:44.239716 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
W20260131 03:08:44.239863 29629 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:44.235551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:36583 (local address 127.19.84.62:34775)
0131 03:08:44.235629 (+ 78us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:44.235632 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:44.235730 (+ 98us) server_negotiation.cc:408] Connection header received
0131 03:08:44.235772 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:44.235773 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:44.235786 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:44.235838 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:44.236270 (+ 432us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:44.236790 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:44.237464 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:44.237577 (+ 113us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:44.239797 (+ 2220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.3:36583: BlockingRecv error: recv got EOF from 127.19.84.3:36583 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jan 31 03:08:44 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:44 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:44.999461 29632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:44.989771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.1:57483)
0131 03:08:44.989985 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:44.989999 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:44.990078 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:44.990343 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:44.990344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:44.990510 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:44.990681 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:44.990683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:44.991483 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:44.991484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:44.991925 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:44.991927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:44.992000 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:44.997160 (+ 5160us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:44.997166 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:44.999333 (+ 2167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":104,"threads_started":1}
W20260131 03:08:44.999701 29633 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:44.989916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:57483 (local address 127.19.84.62:34775)
0131 03:08:44.990051 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:44.990053 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:44.990063 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:08:44.990242 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:44.990243 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:44.990254 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:44.990324 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:44.990779 (+ 455us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:44.991295 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:44.992037 (+ 742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:44.992201 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:44.999636 (+ 7435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.1:57483: BlockingRecv error: recv got EOF from 127.19.84.1:57483 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
Jan 31 03:08:45 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:45 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:45.124778 29634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:45.120486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.2:39539)
0131 03:08:45.120644 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:45.120655 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:45.120706 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:45.120847 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:45.120848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:45.121019 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:45.121174 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:45.121176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:45.121859 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:45.121860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:45.122257 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:45.122258 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:45.122335 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:45.122885 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:45.122890 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:45.124720 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
W20260131 03:08:45.124898 29633 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:45.120608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:39539 (local address 127.19.84.62:34775)
0131 03:08:45.120676 (+ 68us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:45.120678 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:45.120690 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:08:45.120771 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:45.120771 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:45.120782 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:45.120826 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:45.121260 (+ 434us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:45.121766 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:45.122389 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:45.122563 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:45.124820 (+ 2257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.2:39539: BlockingRecv error: recv got EOF from 127.19.84.2:39539 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jan 31 03:08:45 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:45 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:45.244588 29635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:45.240265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.3:43363)
0131 03:08:45.240381 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:45.240393 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:45.240451 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:45.240600 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:45.240601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:45.240779 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:45.240933 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:45.240935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:45.241617 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:45.241618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:45.242116 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:45.242118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:45.242197 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:45.242665 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:45.242669 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:45.244487 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
W20260131 03:08:45.244630 29633 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:45.240372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:43363 (local address 127.19.84.62:34775)
0131 03:08:45.240460 (+ 88us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:45.240462 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:45.240472 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:08:45.240506 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:45.240507 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:45.240524 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:45.240581 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:45.241016 (+ 435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:45.241502 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:45.242198 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:45.242396 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:45.244569 (+ 2173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.3:43363: BlockingRecv error: recv got EOF from 127.19.84.3:43363 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jan 31 03:08:45 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:46 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:46.004294 29636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:45.999978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.1:55927)
0131 03:08:46.000118 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:46.000129 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:46.000179 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:46.000333 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:46.000333 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:46.000518 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:46.000683 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:46.000684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:46.001455 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:46.001457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:46.001846 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:46.001847 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:46.001914 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:46.002446 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:46.002450 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:46.004229 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
W20260131 03:08:46.004480 29637 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:46.000036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:55927 (local address 127.19.84.62:34775)
0131 03:08:46.000171 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:46.000173 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:46.000182 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:08:46.000231 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:46.000232 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:46.000243 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:46.000303 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:46.000780 (+ 477us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:46.001287 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:46.001930 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:46.002117 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:46.004376 (+ 2259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.1:55927: BlockingRecv error: recv got EOF from 127.19.84.1:55927 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":84,"threads_started":1}
Jan 31 03:08:46 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:46 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:46.129729 29638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:46.125389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.2:46525)
0131 03:08:46.125554 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:46.125564 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:46.125621 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:46.125759 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:46.125760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:46.125920 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:46.126076 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:46.126077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:46.126748 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:46.126749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:46.127101 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:46.127102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:46.127161 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:46.127740 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:46.127745 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:46.129670 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
W20260131 03:08:46.129838 29637 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:46.125478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:46525 (local address 127.19.84.62:34775)
0131 03:08:46.125571 (+ 93us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:46.125573 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:46.125623 (+ 50us) server_negotiation.cc:408] Connection header received
0131 03:08:46.125681 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:46.125682 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:46.125694 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:46.125740 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:46.126174 (+ 434us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:46.126674 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:46.127196 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:46.127335 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:46.129765 (+ 2430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.2:46525: BlockingRecv error: recv got EOF from 127.19.84.2:46525 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jan 31 03:08:46 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:46 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:46.249514 29639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:46.245144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.3:57631)
0131 03:08:46.245314 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:46.245330 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:46.245395 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:46.245568 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:46.245568 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:46.245769 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:46.245926 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:46.245928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:46.246605 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:46.246606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:46.246983 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:46.246984 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:46.247071 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:46.247660 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:46.247665 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:46.249421 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":111,"threads_started":1}
W20260131 03:08:46.249572 29637 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:46.245298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:57631 (local address 127.19.84.62:34775)
0131 03:08:46.245379 (+ 81us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:46.245382 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:46.245395 (+ 13us) server_negotiation.cc:408] Connection header received
0131 03:08:46.245473 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:46.245474 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:46.245491 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:46.245559 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:46.246020 (+ 461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:46.246511 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:46.247071 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:46.247268 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:46.249509 (+ 2241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.3:57631: BlockingRecv error: recv got EOF from 127.19.84.3:57631 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jan 31 03:08:47 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:47 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:47.009626 29640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:47.004856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.1:60179)
0131 03:08:47.004989 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:47.005007 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:47.005086 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:47.005282 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:47.005283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:47.005525 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:47.005780 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:47.005783 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:47.006524 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:47.006526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:47.006927 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:47.006928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:47.007027 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:47.007584 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:47.007589 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:47.009516 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
W20260131 03:08:47.009677 29641 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:47.004978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:60179 (local address 127.19.84.62:34775)
0131 03:08:47.005104 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:47.005106 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:47.005116 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:08:47.005162 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:47.005163 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:47.005176 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:47.005266 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:47.005892 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:47.006431 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:47.007025 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:47.007219 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:47.009608 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.1:60179: BlockingRecv error: recv got EOF from 127.19.84.1:60179 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
Jan 31 03:08:47 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:47 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:47.134290 29642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:47.130225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.2:56509)
0131 03:08:47.130366 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:47.130388 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:47.130442 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:47.130549 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:47.130549 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:47.130706 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:47.130863 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:47.130865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:47.131530 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:47.131530 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:47.131905 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:47.131907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:47.131965 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:47.132474 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:47.132479 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:47.134197 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":100,"threads_started":1}
W20260131 03:08:47.134333 29641 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:47.130290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:56509 (local address 127.19.84.62:34775)
0131 03:08:47.130363 (+ 73us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:47.130365 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:47.130442 (+ 77us) server_negotiation.cc:408] Connection header received
0131 03:08:47.130485 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:47.130486 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:47.130496 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:47.130539 (+ 43us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:47.130926 (+ 387us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:47.131458 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:47.131976 (+ 518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:47.132088 (+ 112us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:47.134271 (+ 2183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.2:56509: BlockingRecv error: recv got EOF from 127.19.84.2:56509 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jan 31 03:08:47 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:47 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:47.254660 29643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:47.249994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.3:36805)
0131 03:08:47.250174 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:47.250188 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:47.250295 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:47.250472 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:47.250473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:47.250665 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:47.250823 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:47.250825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:47.251550 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:47.251552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:47.252163 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:47.252166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:47.252261 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:47.252891 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:47.252895 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:47.254580 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":93,"threads_started":1}
W20260131 03:08:47.254765 29641 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:47.250086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:36805 (local address 127.19.84.62:34775)
0131 03:08:47.250177 (+ 91us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:47.250180 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:47.250270 (+ 90us) server_negotiation.cc:408] Connection header received
0131 03:08:47.250390 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:47.250395 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:47.250405 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:47.250465 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:47.250912 (+ 447us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:47.251479 (+ 567us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:47.252283 (+ 804us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:47.252509 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:47.254691 (+ 2182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.3:36805: BlockingRecv error: recv got EOF from 127.19.84.3:36805 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jan 31 03:08:48 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:48 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:48.014559 29644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:48.010187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.1:53161)
0131 03:08:48.010371 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:48.010383 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:48.010445 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:48.010603 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:48.010603 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:48.010773 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:48.010929 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:48.010931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:48.011638 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:48.011639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:48.012090 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:48.012092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:48.012150 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:48.012674 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:48.012678 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:48.014492 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
W20260131 03:08:48.014707 29645 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:48.010294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:53161 (local address 127.19.84.62:34775)
0131 03:08:48.010412 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:48.010414 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:48.010426 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:08:48.010512 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:48.010512 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:48.010522 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:48.010584 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:48.011008 (+ 424us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:48.011545 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:48.012155 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:48.012277 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:48.014612 (+ 2335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.1:53161: BlockingRecv error: recv got EOF from 127.19.84.1:53161 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":63,"threads_started":1}
Jan 31 03:08:48 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:48 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:48.139101 29646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:48.134902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.2:41001)
0131 03:08:48.135035 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:48.135054 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:48.135103 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:48.135264 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:48.135265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:48.135534 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:48.135785 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:48.135788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:48.136451 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:48.136453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:48.136839 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:48.136840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:48.136922 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:48.137381 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:48.137385 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:48.139036 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
W20260131 03:08:48.139437 29645 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:48.135041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:41001 (local address 127.19.84.62:34775)
0131 03:08:48.135109 (+ 68us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:48.135111 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:48.135123 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:08:48.135152 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:48.135153 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:48.135163 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:48.135253 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:48.135872 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:48.136370 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:48.136928 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:48.137134 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:48.139295 (+ 2161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.2:41001: BlockingRecv error: recv got EOF from 127.19.84.2:41001 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jan 31 03:08:48 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:48 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:48.259238 29647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:48.255261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.3:38635)
0131 03:08:48.255410 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:48.255420 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:48.255476 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:48.255592 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:48.255593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:48.255751 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:48.255902 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:48.255904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:48.256546 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:48.256547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:48.256916 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:48.256917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:48.256975 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:48.257458 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:48.257462 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:48.259134 (+ 1672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":78,"threads_started":1}
W20260131 03:08:48.259285 29645 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:48.255301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:38635 (local address 127.19.84.62:34775)
0131 03:08:48.255355 (+ 54us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:48.255357 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:48.255465 (+ 108us) server_negotiation.cc:408] Connection header received
0131 03:08:48.255516 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:48.255517 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:48.255526 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:48.255573 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:48.255969 (+ 396us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:48.256472 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:48.256991 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:48.257110 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:48.259221 (+ 2111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.3:38635: BlockingRecv error: recv got EOF from 127.19.84.3:38635 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jan 31 03:08:49 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:49 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:49.019331 29648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:49.015099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.1:53271)
0131 03:08:49.015257 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:49.015269 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:49.015328 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:49.015525 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:49.015526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:49.015707 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:49.015865 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:49.015866 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:49.016591 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:49.016592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:49.016982 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:49.016984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:49.017043 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:49.017549 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:49.017554 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:49.019273 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
W20260131 03:08:49.019563 29649 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:49.015190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:53271 (local address 127.19.84.62:34775)
0131 03:08:49.015305 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:49.015307 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:49.015319 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:08:49.015436 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:49.015437 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:49.015448 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:49.015512 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:49.015952 (+ 440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:49.016500 (+ 548us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:49.017053 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:49.017167 (+ 114us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:49.019419 (+ 2252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.1:53271: BlockingRecv error: recv got EOF from 127.19.84.1:53271 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":50,"threads_started":1}
Jan 31 03:08:49 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:49 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:49.144454 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:49.139725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.2:35663)
0131 03:08:49.139924 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:49.139937 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:49.140036 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:49.140197 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:49.140198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:49.140406 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:49.140563 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:49.140565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:49.141251 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:49.141252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:49.141703 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:49.141705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:49.141810 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:49.142376 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:49.142380 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:49.144342 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":86,"threads_started":1}
W20260131 03:08:49.144490 29649 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:49.139827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:35663 (local address 127.19.84.62:34775)
0131 03:08:49.139886 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:49.139888 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:49.140015 (+ 127us) server_negotiation.cc:408] Connection header received
0131 03:08:49.140103 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:49.140105 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:49.140119 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:49.140191 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:49.140648 (+ 457us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:49.141148 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:49.141823 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:49.142026 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:49.144430 (+ 2404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.2:35663: BlockingRecv error: recv got EOF from 127.19.84.2:35663 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jan 31 03:08:49 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:49 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:49.264173 29651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:49.259839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.3:49801)
0131 03:08:49.259961 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:49.259976 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:49.260043 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:49.260176 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:49.260177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:49.260365 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:49.260528 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:49.260530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:49.261181 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:49.261182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:49.261581 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:49.261582 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:49.261642 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:49.262112 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:49.262116 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:49.264109 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":66,"threads_started":1}
W20260131 03:08:49.264302 29649 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:49.259925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:49801 (local address 127.19.84.62:34775)
0131 03:08:49.260004 (+ 79us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:49.260006 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:49.260031 (+ 25us) server_negotiation.cc:408] Connection header received
0131 03:08:49.260097 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:49.260098 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:49.260107 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:49.260158 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:49.260616 (+ 458us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:49.261104 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:49.261661 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:49.261779 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:49.264216 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.3:49801: BlockingRecv error: recv got EOF from 127.19.84.3:49801 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jan 31 03:08:50 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:50 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:50.027904 29652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:50.019886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.1:38045)
0131 03:08:50.020042 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:50.020072 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:50.020131 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:50.020303 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:50.020304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:50.020492 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:50.020667 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:50.020669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:50.021458 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:50.021459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:50.021872 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:50.021874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:50.021943 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:50.025315 (+ 3372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:50.025320 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:50.027804 (+ 2484us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
W20260131 03:08:50.028046 29653 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:50.019942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:38045 (local address 127.19.84.62:34775)
0131 03:08:50.020109 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:50.020111 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:50.020121 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:08:50.020186 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:50.020187 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:50.020197 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:50.020265 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:50.020756 (+ 491us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:50.021356 (+ 600us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:50.021976 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:50.022154 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:50.027966 (+ 5812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.1:38045: BlockingRecv error: recv got EOF from 127.19.84.1:38045 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
Jan 31 03:08:50 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:50 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:50.149483 29654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:50.145026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.2:58469)
0131 03:08:50.145211 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:50.145225 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:50.145317 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:50.145481 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:50.145482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:50.145689 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:50.145846 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:50.145848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:50.146551 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:50.146552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:50.146990 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:50.146992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:50.147073 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:50.147594 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:50.147599 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:50.149423 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":98,"threads_started":1}
W20260131 03:08:50.149715 29653 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:50.145130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:58469 (local address 127.19.84.62:34775)
0131 03:08:50.145214 (+ 84us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:50.145217 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:50.145296 (+ 79us) server_negotiation.cc:408] Connection header received
0131 03:08:50.145393 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:50.145394 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:50.145410 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:50.145473 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:50.145936 (+ 463us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:50.146465 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:50.147070 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:50.147274 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:50.149642 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.2:58469: BlockingRecv error: recv got EOF from 127.19.84.2:58469 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jan 31 03:08:50 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:50 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:50.269115 29655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:50.264727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.3:34069)
0131 03:08:50.264886 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:50.264898 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:50.264957 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:50.265094 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:50.265095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:50.265260 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:50.265432 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:50.265434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:50.266051 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:50.266052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:50.266478 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:50.266480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:50.266563 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:50.267179 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:50.267184 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:50.269058 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":105,"threads_started":1}
W20260131 03:08:50.269299 29653 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:50.264804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:34069 (local address 127.19.84.62:34775)
0131 03:08:50.264885 (+ 81us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:50.264887 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:50.264956 (+ 69us) server_negotiation.cc:408] Connection header received
0131 03:08:50.265021 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:50.265022 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:50.265032 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:50.265076 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:50.265499 (+ 423us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:50.265974 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:50.266577 (+ 603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:50.266777 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:50.269238 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.3:34069: BlockingRecv error: recv got EOF from 127.19.84.3:34069 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jan 31 03:08:51 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:51 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:51.034253 29656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:51.028592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.1:46011)
0131 03:08:51.028740 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:51.028752 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:51.028809 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:51.028986 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:51.028987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:51.029223 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:51.029524 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:51.029527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:51.030613 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:51.030615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:51.031215 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:51.031217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:51.031290 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:51.031850 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:51.031856 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:51.034160 (+ 2304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
W20260131 03:08:51.034313 29657 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:51.028690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:46011 (local address 127.19.84.62:34775)
0131 03:08:51.028821 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:51.028824 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:51.028836 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:08:51.028868 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:51.028869 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:51.028883 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:51.028969 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:51.029632 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:51.030513 (+ 881us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:51.031295 (+ 782us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:51.031531 (+ 236us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:51.034247 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.1:46011: BlockingRecv error: recv got EOF from 127.19.84.1:46011 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":56,"threads_started":1}
Jan 31 03:08:51 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:51 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:51.157626 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:51.150022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.2:53219)
0131 03:08:51.150210 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:51.150223 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:51.150313 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:51.150528 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:51.150530 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:51.150717 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:51.150872 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:51.150874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:51.151802 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:51.151803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:51.152178 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:51.152179 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:51.152262 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:51.155687 (+ 3425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:51.155692 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:51.157533 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":102,"threads_started":1}
W20260131 03:08:51.157699 29657 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:51.150115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:53219 (local address 127.19.84.62:34775)
0131 03:08:51.150202 (+ 87us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:51.150205 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:51.150293 (+ 88us) server_negotiation.cc:408] Connection header received
0131 03:08:51.150437 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:51.150438 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:51.150451 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:51.150521 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:51.150961 (+ 440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:51.151722 (+ 761us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:51.152259 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:51.152454 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:51.157635 (+ 5181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.2:53219: BlockingRecv error: recv got EOF from 127.19.84.2:53219 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jan 31 03:08:51 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:51 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:51.274534 29659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:51.269647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.3:57757)
0131 03:08:51.269819 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:51.269834 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:51.269964 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:51.270242 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:51.270244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:51.270433 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:51.270597 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:51.270599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:51.271241 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:51.271242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:51.271763 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:51.271765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:51.271846 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:51.272486 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:51.272491 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:51.274477 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":85,"threads_started":1}
W20260131 03:08:51.274689 29657 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:51.269752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:57757 (local address 127.19.84.62:34775)
0131 03:08:51.269824 (+ 72us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:51.269827 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:51.269942 (+ 115us) server_negotiation.cc:408] Connection header received
0131 03:08:51.270037 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:51.270038 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:51.270050 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:51.270230 (+ 180us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:51.270678 (+ 448us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:51.271164 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:51.271845 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:51.272037 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:51.274605 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.3:57757: BlockingRecv error: recv got EOF from 127.19.84.3:57757 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jan 31 03:08:52 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:52 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:52.039440 29660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:52.034735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.1:55581)
0131 03:08:52.034885 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:52.034899 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:52.034979 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:52.035161 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:52.035162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:52.035358 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:52.035589 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:52.035591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:52.036242 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:52.036243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:52.036713 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:52.036715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:52.036798 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:52.037511 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:52.037515 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:52.039287 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
W20260131 03:08:52.039613 29661 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:52.034789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:55581 (local address 127.19.84.62:34775)
0131 03:08:52.034939 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:52.034942 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:52.034954 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:08:52.035049 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:52.035050 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:52.035067 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:52.035145 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:52.035669 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:52.036170 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:52.036979 (+ 809us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:52.037184 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:52.039505 (+ 2321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.1:55581: BlockingRecv error: recv got EOF from 127.19.84.1:55581 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
Jan 31 03:08:52 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:52 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:52.162575 29662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:52.158210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.2:52035)
0131 03:08:52.158380 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:52.158398 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:52.158476 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:52.158647 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:52.158656 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:52.158839 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:52.158995 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:52.158998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:52.159706 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:52.159708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:52.160088 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:52.160090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:52.160157 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:52.160703 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:52.160708 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:52.162467 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":111,"threads_started":1}
W20260131 03:08:52.162606 29661 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:52.158262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:52035 (local address 127.19.84.62:34775)
0131 03:08:52.158338 (+ 76us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:52.158340 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:52.158463 (+ 123us) server_negotiation.cc:408] Connection header received
0131 03:08:52.158552 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:52.158553 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:52.158567 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:52.158628 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:52.159069 (+ 441us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:52.159628 (+ 559us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:52.160164 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:52.160281 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:52.162549 (+ 2268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.2:52035: BlockingRecv error: recv got EOF from 127.19.84.2:52035 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jan 31 03:08:52 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:52 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:52.279668 29663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:52.275033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.3:50057)
0131 03:08:52.275191 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:52.275202 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:52.275262 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:52.275468 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:52.275469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:52.275647 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:52.275804 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:52.275805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:52.276513 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:52.276515 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:52.276885 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:52.276887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:52.276952 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:52.277542 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:52.277546 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:52.279569 (+ 2023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
W20260131 03:08:52.279718 29661 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:52.275143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:50057 (local address 127.19.84.62:34775)
0131 03:08:52.275205 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:52.275207 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:52.275248 (+ 41us) server_negotiation.cc:408] Connection header received
0131 03:08:52.275324 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:52.275325 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:52.275335 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:52.275452 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:52.275898 (+ 446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:52.276414 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:52.276957 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:52.277067 (+ 110us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:52.279651 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.3:50057: BlockingRecv error: recv got EOF from 127.19.84.3:50057 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jan 31 03:08:53 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:53 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:53.044318 29664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:53.039931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.1:46855)
0131 03:08:53.040080 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:53.040091 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:53.040149 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:53.040278 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:53.040279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:53.040461 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:53.040626 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:53.040628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:53.041252 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:53.041254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:53.041823 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:53.041825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:53.041885 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:53.042388 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:53.042392 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:53.044222 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
W20260131 03:08:53.044382 29665 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:53.040027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:46855 (local address 127.19.84.62:34775)
0131 03:08:53.040146 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:53.040148 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:53.040158 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:08:53.040198 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:53.040199 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:53.040209 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:53.040267 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:53.040690 (+ 423us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:53.041186 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:53.041895 (+ 709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:53.042035 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:53.044314 (+ 2279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.1:46855: BlockingRecv error: recv got EOF from 127.19.84.1:46855 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":67,"threads_started":1}
Jan 31 03:08:53 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:53 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:53.167326 29666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:53.163095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.2:38775)
0131 03:08:53.163300 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:53.163312 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:53.163409 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:53.163547 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:53.163548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:53.163738 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:53.163902 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:53.163903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:53.164550 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:53.164551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:53.164924 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:53.164926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:53.164983 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:53.165482 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:53.165487 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:53.167220 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":85,"threads_started":1}
W20260131 03:08:53.167368 29665 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:53.163205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:38775 (local address 127.19.84.62:34775)
0131 03:08:53.163249 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:53.163252 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:53.163411 (+ 159us) server_negotiation.cc:408] Connection header received
0131 03:08:53.163463 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:53.163464 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:53.163475 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:53.163535 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:53.163982 (+ 447us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:53.164479 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:53.164997 (+ 518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:53.165165 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:53.167306 (+ 2141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.2:38775: BlockingRecv error: recv got EOF from 127.19.84.2:38775 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Jan 31 03:08:53 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:53 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:53.284838 29667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:53.280352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.3:43187)
0131 03:08:53.280535 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:53.280548 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:53.280656 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:53.280809 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:53.280810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:53.281005 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:53.281184 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:53.281186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:53.281869 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:53.281870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:53.282252 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:53.282254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:53.282337 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:53.282870 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:53.282875 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:53.284744 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":111,"threads_started":1}
W20260131 03:08:53.284902 29665 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:53.280566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:43187 (local address 127.19.84.62:34775)
0131 03:08:53.280636 (+ 70us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:53.280639 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:53.280650 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:08:53.280720 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:53.280721 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:53.280735 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:53.280801 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:53.281274 (+ 473us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:53.281780 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:53.282340 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:53.282525 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:53.284839 (+ 2314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.3:43187: BlockingRecv error: recv got EOF from 127.19.84.3:43187 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jan 31 03:08:54 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:54 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:54.049352 29668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:54.044826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.1:46403)
0131 03:08:54.044958 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:54.044968 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:54.045018 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:54.045178 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:54.045179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:54.045362 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:54.045516 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:54.045518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:54.046176 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:54.046177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:54.046595 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:54.046597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:54.046659 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:54.047288 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:54.047293 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:54.049291 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
W20260131 03:08:54.049580 29669 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:54.044896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:46403 (local address 127.19.84.62:34775)
0131 03:08:54.045058 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:54.045060 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:54.045069 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:08:54.045091 (+ 22us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:54.045092 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:54.045101 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:54.045166 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:54.045589 (+ 423us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:54.046102 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:54.046683 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:54.046874 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:54.049493 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.1:46403: BlockingRecv error: recv got EOF from 127.19.84.1:46403 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
Jan 31 03:08:54 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:54 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:54.172173 29670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:54.167819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.2:53759)
0131 03:08:54.167952 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:54.167963 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:54.168022 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:54.168170 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:54.168171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:54.168349 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:54.168503 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:54.168505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:54.169175 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:54.169176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:54.169586 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:54.169588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:54.169664 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:54.170161 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:54.170166 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:54.172113 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
W20260131 03:08:54.172314 29669 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:54.167925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:53759 (local address 127.19.84.62:34775)
0131 03:08:54.167992 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:54.167994 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:54.168007 (+ 13us) server_negotiation.cc:408] Connection header received
0131 03:08:54.168080 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:54.168081 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:54.168092 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:54.168149 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:54.168578 (+ 429us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:54.169082 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:54.169689 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:54.169877 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:54.172232 (+ 2355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.2:53759: BlockingRecv error: recv got EOF from 127.19.84.2:53759 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jan 31 03:08:54 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:54 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:54.289929 29671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:54.285525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.3:46351)
0131 03:08:54.285688 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:54.285705 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:54.285782 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:54.285974 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:54.285975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:54.286196 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:54.286526 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:54.286539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:54.287195 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:54.287196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:54.287647 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:54.287649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:54.287713 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:54.288198 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:54.288202 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:54.289868 (+ 1666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":71,"threads_started":1}
W20260131 03:08:54.290104 29669 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:54.285638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:46351 (local address 127.19.84.62:34775)
0131 03:08:54.285714 (+ 76us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:54.285717 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:54.285853 (+ 136us) server_negotiation.cc:408] Connection header received
0131 03:08:54.285880 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:54.285881 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:54.285895 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:54.285948 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:54.286625 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:54.287109 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:54.287737 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:54.287860 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:54.290035 (+ 2175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.3:46351: BlockingRecv error: recv got EOF from 127.19.84.3:46351 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jan 31 03:08:55 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:55 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:55.058939 29672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:55.049912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.1:34311)
0131 03:08:55.050088 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:55.050107 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:55.050163 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:55.050375 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:55.050377 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:55.050660 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:55.050901 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:55.050904 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:55.051729 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:55.051730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:55.052170 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:55.052172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:55.052277 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:55.057015 (+ 4738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:55.057020 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:55.058880 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":121,"threads_started":1}
W20260131 03:08:55.059144 29673 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:55.050012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:34311 (local address 127.19.84.62:34775)
0131 03:08:55.050146 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:55.050148 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:55.050159 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:08:55.050223 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:55.050224 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:55.050235 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:55.050339 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:55.051008 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:55.051650 (+ 642us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:55.052282 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:55.052422 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:55.059082 (+ 6660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.1:34311: BlockingRecv error: recv got EOF from 127.19.84.1:34311 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
Jan 31 03:08:55 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:55 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:55.177596 29674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:55.172846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.2:48059)
0131 03:08:55.173080 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:55.173094 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:55.173199 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:55.173524 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:55.173526 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:55.173759 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:55.173919 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:55.173922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:55.174621 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:55.174622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:55.175005 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:55.175006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:55.175074 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:55.175682 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:55.175686 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:55.177537 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":112,"threads_started":1}
W20260131 03:08:55.177798 29673 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:55.172936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:48059 (local address 127.19.84.62:34775)
0131 03:08:55.173060 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:55.173062 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:55.173168 (+ 106us) server_negotiation.cc:408] Connection header received
0131 03:08:55.173274 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:55.173275 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:55.173288 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:55.173461 (+ 173us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:55.174010 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:55.174508 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:55.175086 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:55.175219 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:55.177738 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.2:48059: BlockingRecv error: recv got EOF from 127.19.84.2:48059 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jan 31 03:08:55 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:55 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:55.294867 29675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:55.290449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.3:50791)
0131 03:08:55.290612 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:55.290628 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:55.290688 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:55.290846 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:55.290847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:55.291013 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:55.291170 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:55.291171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:55.291905 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:55.291907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:55.292305 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:55.292307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:55.292376 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:55.292918 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:55.292923 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:55.294798 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":116,"threads_started":1}
W20260131 03:08:55.294992 29673 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:55.290538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:50791 (local address 127.19.84.62:34775)
0131 03:08:55.290619 (+ 81us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:55.290621 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:55.290740 (+ 119us) server_negotiation.cc:408] Connection header received
0131 03:08:55.290770 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:55.290771 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:55.290781 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:55.290828 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:55.291271 (+ 443us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:55.291805 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:55.292397 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:55.292527 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:55.294916 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.3:50791: BlockingRecv error: recv got EOF from 127.19.84.3:50791 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jan 31 03:08:56 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:56 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:56.063664 29676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:56.059503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.1:52671)
0131 03:08:56.059655 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:56.059672 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:56.059739 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:56.059880 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:56.059881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:56.060048 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:56.060206 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:56.060208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:56.060858 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:56.060859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:56.061226 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:56.061227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:56.061287 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:56.061750 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:56.061754 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:56.063606 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
W20260131 03:08:56.063830 29677 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:56.059569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:52671 (local address 127.19.84.62:34775)
0131 03:08:56.059713 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:56.059715 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:56.059726 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:08:56.059795 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:56.059796 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:56.059806 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:56.059865 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:56.060267 (+ 402us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:56.060789 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:56.061295 (+ 506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:56.061427 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:56.063757 (+ 2330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.1:52671: BlockingRecv error: recv got EOF from 127.19.84.1:52671 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
Jan 31 03:08:56 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:56 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:56.182430 29678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:56.178073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.2:42461)
0131 03:08:56.178209 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:56.178221 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:56.178287 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:56.178484 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:56.178486 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:56.178683 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:56.178842 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:56.178844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:56.179591 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:56.179593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:56.179970 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:56.179971 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:56.180051 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:56.180550 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:56.180555 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:56.182329 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
W20260131 03:08:56.182498 29677 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:56.178245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:42461 (local address 127.19.84.62:34775)
0131 03:08:56.178318 (+ 73us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:56.178320 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:56.178329 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:08:56.178393 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:56.178395 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:56.178411 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:56.178474 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:56.178935 (+ 461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:56.179494 (+ 559us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:56.180061 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:56.180261 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:56.182420 (+ 2159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.2:42461: BlockingRecv error: recv got EOF from 127.19.84.2:42461 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jan 31 03:08:56 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:56 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:56.300165 29679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:56.295486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.3:58343)
0131 03:08:56.295636 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:56.295646 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:56.295708 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:56.295863 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:56.295864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:56.296028 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:56.296187 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:56.296189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:56.296970 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:56.296971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:56.297516 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:56.297519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:56.297586 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:56.298213 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:56.298219 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:56.300106 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
W20260131 03:08:56.300331 29677 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:56.295576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:58343 (local address 127.19.84.62:34775)
0131 03:08:56.295658 (+ 82us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:56.295660 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:56.295692 (+ 32us) server_negotiation.cc:408] Connection header received
0131 03:08:56.295766 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:56.295767 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:56.295777 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:56.295845 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:56.296265 (+ 420us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:56.296903 (+ 638us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:56.297599 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:56.297720 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:56.300256 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.3:58343: BlockingRecv error: recv got EOF from 127.19.84.3:58343 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jan 31 03:08:57 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:57 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:57.068681 29680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:57.064159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.1:52633)
0131 03:08:57.064322 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:57.064334 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:57.064386 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:57.064533 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:57.064534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:57.064709 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:57.064867 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:57.064869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:57.065614 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:57.065615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:57.065995 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:57.065997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:57.066060 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:57.066598 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:57.066602 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:57.068621 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":99,"threads_started":1}
W20260131 03:08:57.068872 29681 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:57.064258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:52633 (local address 127.19.84.62:34775)
0131 03:08:57.064358 (+ 100us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:57.064360 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:57.064373 (+ 13us) server_negotiation.cc:408] Connection header received
0131 03:08:57.064444 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:57.064445 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:57.064456 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:57.064521 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:57.064966 (+ 445us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:57.065518 (+ 552us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:57.066095 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:57.066306 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:57.068805 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.1:52633: BlockingRecv error: recv got EOF from 127.19.84.1:52633 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":50,"threads_started":1}
Jan 31 03:08:57 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:57 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:57.188158 29682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:57.183127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.2:32907)
0131 03:08:57.183268 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:57.183281 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:57.183418 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:57.183611 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:57.183612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:57.183859 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:57.184094 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:57.184098 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:57.184844 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:57.184846 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:57.185232 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:57.185234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:57.185329 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:57.185948 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:57.185952 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:57.188097 (+ 2145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
W20260131 03:08:57.188320 29681 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:57.183231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:32907 (local address 127.19.84.62:34775)
0131 03:08:57.183294 (+ 63us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:57.183297 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:57.183388 (+ 91us) server_negotiation.cc:408] Connection header received
0131 03:08:57.183496 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:57.183497 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:57.183512 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:57.183585 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:57.184169 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:57.184750 (+ 581us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:57.185322 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:57.185562 (+ 240us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:57.188232 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.2:32907: BlockingRecv error: recv got EOF from 127.19.84.2:32907 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jan 31 03:08:57 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:57 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.3@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:57.305661 29683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:57.300791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.3:50877)
0131 03:08:57.300959 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:57.300977 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:57.301050 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:57.301200 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:57.301201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:57.301390 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:57.301568 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:57.301570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:57.302278 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:57.302279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:57.302681 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:57.302683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:57.302761 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:57.303521 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:57.303526 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:57.305561 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":74,"threads_started":1}
W20260131 03:08:57.305823 29681 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:57.300910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:50877 (local address 127.19.84.62:34775)
0131 03:08:57.300974 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:57.300976 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:57.301042 (+ 66us) server_negotiation.cc:408] Connection header received
0131 03:08:57.301098 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:57.301099 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:57.301109 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:57.301161 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:57.301645 (+ 484us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:57.302199 (+ 554us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:57.302782 (+ 583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:57.302917 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:57.305740 (+ 2823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.3:50877: BlockingRecv error: recv got EOF from 127.19.84.3:50877 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jan 31 03:08:58 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:58 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.1@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:58.073472 29684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:58.069285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.1:33181)
0131 03:08:58.069436 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:58.069449 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:58.069526 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:58.069652 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:58.069652 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:58.069821 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:58.069978 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:58.069980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:58.070684 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:58.070685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:58.071065 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:58.071066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:58.071127 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:58.071724 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:58.071728 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:58.073408 (+ 1680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
W20260131 03:08:58.073625 29685 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:58.069392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:33181 (local address 127.19.84.62:34775)
0131 03:08:58.069525 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:58.069528 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:58.069538 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:08:58.069567 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:58.069568 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:58.069579 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:58.069639 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:58.070066 (+ 427us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:58.070608 (+ 542us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:58.071224 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:58.071390 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:58.073540 (+ 2150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.1:33181: BlockingRecv error: recv got EOF from 127.19.84.1:33181 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
Jan 31 03:08:58 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
Jan 31 03:08:58 dist-test-slave-5nnp krb5kdc[28618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.84.2@KRBTEST.COM for oryx/127.19.84.62@KRBTEST.COM, Server not found in Kerberos database
I20260131 03:08:58.193019 29686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:58.188707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34775 (local address 127.19.84.2:40227)
0131 03:08:58.188852 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:58.188864 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:58.188927 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:58.189069 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:58.189069 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:58.189236 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:58.189401 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:58.189403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:58.190035 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:58.190036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:58.190433 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:58.190434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:58.190494 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:58.191123 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:08:58.191128 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:58.192931 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.84.62:34775: Server oryx/127.19.84.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
W20260131 03:08:58.193094 29685 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:58.188811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:40227 (local address 127.19.84.62:34775)
0131 03:08:58.188876 (+ 65us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:58.188878 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:58.188909 (+ 31us) server_negotiation.cc:408] Connection header received
0131 03:08:58.188991 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:58.188992 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:58.189003 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:58.189050 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:58.189465 (+ 415us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:58.189965 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:58.190503 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:58.190637 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:58.193029 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.84.2:40227: BlockingRecv error: recv got EOF from 127.19.84.2:40227 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260131 03:08:58.228349 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 29210
I20260131 03:08:58.233068 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 29362
I20260131 03:08:58.237689 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 29494
I20260131 03:08:58.242311 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 29140
2026-01-31T03:08:58Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18470 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-01-31T03:08:58Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:08:58Z Disabled control of system clock
I20260131 03:08:58.270277 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:38303
--webserver_interface=127.19.84.62
--webserver_port=0
--builtin_ntp_servers=127.19.84.20:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:38303
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260131 03:08:58.351879 29690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:58.352113 29690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:58.352149 29690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:58.353824 29690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:08:58.353931 29690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:58.353955 29690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:58.353976 29690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:08:58.354033 29690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:08:58.355707 29690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:38303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:38303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:58.356000 29690 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:58.356267 29690 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260131 03:08:58.359529 29690 server_base.cc:1060] running on GCE node
W20260131 03:08:58.359494 29695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:58.359477 29696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:58.359494 29698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:58.359966 29690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:58.360225 29690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:58.361366 29690 hybrid_clock.cc:648] HybridClock initialized: now 1769828938361308 us; error 73 us; skew 500 ppm
I20260131 03:08:58.362600 29690 webserver.cc:492] Webserver started at http://127.19.84.62:40407/ using document root <none> and password file <none>
I20260131 03:08:58.362838 29690 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:58.362881 29690 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:58.363005 29690 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:58.364096 29690 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "9e04ea5c2e52445a975afe01980fe515"
format_stamp: "Formatted at 2026-01-31 03:08:58 on dist-test-slave-5nnp"
server_key: "9ec9742e597c9ca1a06cdb4f91157404"
server_key_iv: "65d68f799874ccb5e8d0f84362a459a8"
server_key_version: "encryptionkey@0"
I20260131 03:08:58.364435 29690 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "9e04ea5c2e52445a975afe01980fe515"
format_stamp: "Formatted at 2026-01-31 03:08:58 on dist-test-slave-5nnp"
server_key: "9ec9742e597c9ca1a06cdb4f91157404"
server_key_iv: "65d68f799874ccb5e8d0f84362a459a8"
server_key_version: "encryptionkey@0"
I20260131 03:08:58.365643 29690 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:08:58.366446 29704 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:58.366654 29690 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:58.366743 29690 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "9e04ea5c2e52445a975afe01980fe515"
format_stamp: "Formatted at 2026-01-31 03:08:58 on dist-test-slave-5nnp"
server_key: "9ec9742e597c9ca1a06cdb4f91157404"
server_key_iv: "65d68f799874ccb5e8d0f84362a459a8"
server_key_version: "encryptionkey@0"
I20260131 03:08:58.366822 29690 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:58.378922 29690 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:58.379227 29690 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:58.379362 29690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:58.383602 29690 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:38303
I20260131 03:08:58.383666 29756 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:38303 every 8 connection(s)
I20260131 03:08:58.384013 29690 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:08:58.384359 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 29690
I20260131 03:08:58.384516 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:08:58.384706 19792 external_mini_cluster.cc:1468] Setting key b4e35e047356b68b8a46f165bb3f5e2e
I20260131 03:08:58.384789 29757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:58.387326 29757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e04ea5c2e52445a975afe01980fe515: Bootstrap starting.
I20260131 03:08:58.387945 29757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e04ea5c2e52445a975afe01980fe515: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:58.388268 29757 log.cc:826] T 00000000000000000000000000000000 P 9e04ea5c2e52445a975afe01980fe515: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:58.389076 29757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e04ea5c2e52445a975afe01980fe515: No bootstrap required, opened a new log
I20260131 03:08:58.389331 29760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:58.385251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56346 (local address 127.19.84.62:38303)
0131 03:08:58.385482 (+ 231us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:58.385488 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:58.385505 (+ 17us) server_negotiation.cc:408] Connection header received
0131 03:08:58.385999 (+ 494us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:58.386004 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:58.386053 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:58.386192 (+ 139us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:58.386581 (+ 389us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:58.387307 (+ 726us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:58.387930 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:58.388187 (+ 257us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:58.388666 (+ 479us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:58.388673 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:58.388673 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:08:58.388866 (+ 193us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:58.388905 (+ 39us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:08:58.388914 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:58.389000 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:58.389062 (+ 62us) server_negotiation.cc:300] Negotiation successful
0131 03:08:58.389124 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
I20260131 03:08:58.390278 29757 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9e04ea5c2e52445a975afe01980fe515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e04ea5c2e52445a975afe01980fe515" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 38303 } }
I20260131 03:08:58.390396 29757 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9e04ea5c2e52445a975afe01980fe515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:58.390416 29757 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9e04ea5c2e52445a975afe01980fe515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e04ea5c2e52445a975afe01980fe515, State: Initialized, Role: FOLLOWER
I20260131 03:08:58.390555 29757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e04ea5c2e52445a975afe01980fe515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e04ea5c2e52445a975afe01980fe515" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 38303 } }
I20260131 03:08:58.390645 29757 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9e04ea5c2e52445a975afe01980fe515 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:08:58.390687 29757 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9e04ea5c2e52445a975afe01980fe515 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:08:58.390736 29757 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9e04ea5c2e52445a975afe01980fe515 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:58.391255 29757 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9e04ea5c2e52445a975afe01980fe515 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e04ea5c2e52445a975afe01980fe515" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 38303 } }
I20260131 03:08:58.391450 29757 leader_election.cc:304] T 00000000000000000000000000000000 P 9e04ea5c2e52445a975afe01980fe515 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e04ea5c2e52445a975afe01980fe515; no voters:
I20260131 03:08:58.391652 29757 leader_election.cc:290] T 00000000000000000000000000000000 P 9e04ea5c2e52445a975afe01980fe515 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:08:58.391773 29762 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9e04ea5c2e52445a975afe01980fe515 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:08:58.391913 29757 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9e04ea5c2e52445a975afe01980fe515 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:08:58.391917 29762 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9e04ea5c2e52445a975afe01980fe515 [term 1 LEADER]: Becoming Leader. State: Replica: 9e04ea5c2e52445a975afe01980fe515, State: Running, Role: LEADER
I20260131 03:08:58.392028 29762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e04ea5c2e52445a975afe01980fe515 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e04ea5c2e52445a975afe01980fe515" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 38303 } }
I20260131 03:08:58.392444 29764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e04ea5c2e52445a975afe01980fe515 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e04ea5c2e52445a975afe01980fe515. Latest consensus state: current_term: 1 leader_uuid: "9e04ea5c2e52445a975afe01980fe515" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e04ea5c2e52445a975afe01980fe515" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 38303 } } }
I20260131 03:08:58.392560 29764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e04ea5c2e52445a975afe01980fe515 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:58.392757 29771 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:08:58.393218 29771 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:08:58.393256 29763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e04ea5c2e52445a975afe01980fe515 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e04ea5c2e52445a975afe01980fe515" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e04ea5c2e52445a975afe01980fe515" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 38303 } } }
I20260131 03:08:58.393504 29763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e04ea5c2e52445a975afe01980fe515 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:58.394659 29771 catalog_manager.cc:1357] Generated new cluster ID: 45b586681c7c4082a74ad723cfaef844
I20260131 03:08:58.394702 29771 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:08:58.399735 29771 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:08:58.400243 29771 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:08:58.404783 29771 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 9e04ea5c2e52445a975afe01980fe515: Generated new TSK 0
I20260131 03:08:58.404964 29771 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:08:58.410269 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:38303
--builtin_ntp_servers=127.19.84.20:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260131 03:08:58.498548 29781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:58.498791 29781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:58.498827 29781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:58.500468 29781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:58.500559 29781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:58.500600 29781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:08:58.502267 29781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:38303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:58.502735 29781 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:58.503093 29781 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260131 03:08:58.506356 29781 server_base.cc:1060] running on GCE node
W20260131 03:08:58.506313 29787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:58.506338 29786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:58.506551 29789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:58.506821 29781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:58.507171 29781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:58.508363 29781 hybrid_clock.cc:648] HybridClock initialized: now 1769828938508340 us; error 37 us; skew 500 ppm
I20260131 03:08:58.509687 29781 webserver.cc:492] Webserver started at http://127.19.84.1:35985/ using document root <none> and password file <none>
I20260131 03:08:58.509905 29781 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:58.509945 29781 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:58.510066 29781 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:58.511072 29781 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "8bc84bbe60e44abdb331f46592ea2860"
format_stamp: "Formatted at 2026-01-31 03:08:58 on dist-test-slave-5nnp"
server_key: "0fa2bb339e07237521fc1d82a3288d5d"
server_key_iv: "9aeb41518a693556424101da5efe371a"
server_key_version: "encryptionkey@0"
I20260131 03:08:58.511463 29781 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "8bc84bbe60e44abdb331f46592ea2860"
format_stamp: "Formatted at 2026-01-31 03:08:58 on dist-test-slave-5nnp"
server_key: "0fa2bb339e07237521fc1d82a3288d5d"
server_key_iv: "9aeb41518a693556424101da5efe371a"
server_key_version: "encryptionkey@0"
I20260131 03:08:58.512706 29781 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:08:58.513522 29795 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:58.513698 29781 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260131 03:08:58.513773 29781 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "8bc84bbe60e44abdb331f46592ea2860"
format_stamp: "Formatted at 2026-01-31 03:08:58 on dist-test-slave-5nnp"
server_key: "0fa2bb339e07237521fc1d82a3288d5d"
server_key_iv: "9aeb41518a693556424101da5efe371a"
server_key_version: "encryptionkey@0"
I20260131 03:08:58.513867 29781 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:58.555887 29781 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:58.556155 29781 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:58.556303 29781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:58.556576 29781 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:58.556993 29781 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:58.557029 29781 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:58.557053 29781 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:58.557085 29781 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:58.563989 29781 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:46719
I20260131 03:08:58.564117 29908 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:46719 every 8 connection(s)
I20260131 03:08:58.564409 29781 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
I20260131 03:08:58.565774 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 29781
I20260131 03:08:58.565884 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:08:58.566079 19792 external_mini_cluster.cc:1468] Setting key 25889119b42d095f0bd637a88902a777
I20260131 03:08:58.567157 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:38303
--builtin_ntp_servers=127.19.84.20:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260131 03:08:58.569818 29760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:58.565458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:45901 (local address 127.19.84.62:38303)
0131 03:08:58.565529 (+ 71us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:58.565531 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:58.565769 (+ 238us) server_negotiation.cc:408] Connection header received
0131 03:08:58.566667 (+ 898us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:58.566669 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:58.566679 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:58.566778 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:58.567304 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:58.567890 (+ 586us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:58.568643 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:58.568793 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:58.569481 (+ 688us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:58.569485 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:58.569486 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:08:58.569491 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:58.569513 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:08:58.569517 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:58.569594 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:58.569701 (+ 107us) server_negotiation.cc:300] Negotiation successful
0131 03:08:58.569740 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260131 03:08:58.569994 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:58.565407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:38303 (local address 127.19.84.1:45901)
0131 03:08:58.565664 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:58.565690 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:58.566492 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:58.566874 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:58.566879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:58.566891 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:58.567201 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:58.567203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:58.567983 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:58.567984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:58.568540 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:58.568543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:58.568701 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:58.569356 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:08:58.569356 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:58.569378 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:08:58.569397 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:58.569609 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:58.569616 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:58.569624 (+ 8us) client_negotiation.cc:769] Sending connection context
0131 03:08:58.569707 (+ 83us) client_negotiation.cc:240] Negotiation successful
0131 03:08:58.569790 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":113,"threads_started":1}
I20260131 03:08:58.570482 29909 heartbeater.cc:344] Connected to a master server at 127.19.84.62:38303
I20260131 03:08:58.570587 29909 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:58.570806 29909 heartbeater.cc:507] Master 127.19.84.62:38303 requested a full tablet report, sending...
I20260131 03:08:58.571349 29721 ts_manager.cc:194] Registered new tserver with Master: 8bc84bbe60e44abdb331f46592ea2860 (127.19.84.1:46719)
I20260131 03:08:58.572455 29721 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.19.84.1:45901
W20260131 03:08:58.647714 29912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:58.647881 29912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:58.647902 29912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:58.649262 29912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:58.649300 29912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:58.649324 29912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:08:58.650887 29912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:38303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:58.651067 29912 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:58.651260 29912 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:58.653889 29920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:58.654032 29918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:58.654048 29912 server_base.cc:1060] running on GCE node
W20260131 03:08:58.653884 29917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:58.654379 29912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:58.654614 29912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:58.655783 29912 hybrid_clock.cc:648] HybridClock initialized: now 1769828938655769 us; error 27 us; skew 500 ppm
I20260131 03:08:58.656967 29912 webserver.cc:492] Webserver started at http://127.19.84.2:45991/ using document root <none> and password file <none>
I20260131 03:08:58.657187 29912 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:58.657251 29912 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:58.657397 29912 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:58.658326 29912 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "29e2df59f1c94c969d4efbb0663efff1"
format_stamp: "Formatted at 2026-01-31 03:08:58 on dist-test-slave-5nnp"
server_key: "4710cd2f06907fbd373d920164117bc6"
server_key_iv: "b67526301db36bbdff2d0711f9412ebe"
server_key_version: "encryptionkey@0"
I20260131 03:08:58.658651 29912 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "29e2df59f1c94c969d4efbb0663efff1"
format_stamp: "Formatted at 2026-01-31 03:08:58 on dist-test-slave-5nnp"
server_key: "4710cd2f06907fbd373d920164117bc6"
server_key_iv: "b67526301db36bbdff2d0711f9412ebe"
server_key_version: "encryptionkey@0"
I20260131 03:08:58.660048 29912 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260131 03:08:58.660822 29926 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:58.661013 29912 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260131 03:08:58.661093 29912 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "29e2df59f1c94c969d4efbb0663efff1"
format_stamp: "Formatted at 2026-01-31 03:08:58 on dist-test-slave-5nnp"
server_key: "4710cd2f06907fbd373d920164117bc6"
server_key_iv: "b67526301db36bbdff2d0711f9412ebe"
server_key_version: "encryptionkey@0"
I20260131 03:08:58.661170 29912 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:58.671952 29912 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:58.672205 29912 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:58.672335 29912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:58.672524 29912 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:58.672825 29912 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:58.672856 29912 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:58.672914 29912 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:58.672950 29912 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:58.678676 29912 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:46125
I20260131 03:08:58.678759 30039 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:46125 every 8 connection(s)
I20260131 03:08:58.679075 29912 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
I20260131 03:08:58.682035 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 29912
I20260131 03:08:58.682147 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:08:58.682341 19792 external_mini_cluster.cc:1468] Setting key 6d3ae7052cba55971d17b82b4e3b51ec
I20260131 03:08:58.683667 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:38303
--builtin_ntp_servers=127.19.84.20:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260131 03:08:58.684641 29760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:58.679878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:47409 (local address 127.19.84.62:38303)
0131 03:08:58.679984 (+ 106us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:58.679987 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:58.680150 (+ 163us) server_negotiation.cc:408] Connection header received
0131 03:08:58.680695 (+ 545us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:58.680697 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:58.680714 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:58.680769 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:58.681245 (+ 476us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:58.681744 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:58.682523 (+ 779us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:58.682818 (+ 295us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:58.684244 (+ 1426us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:58.684248 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:58.684249 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:08:58.684254 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:58.684269 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:08:58.684273 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:58.684301 (+ 28us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:58.684460 (+ 159us) server_negotiation.cc:300] Negotiation successful
0131 03:08:58.684538 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260131 03:08:58.684731 30042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:58.679801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:38303 (local address 127.19.84.2:47409)
0131 03:08:58.680047 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:58.680072 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:58.680583 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:58.680846 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:58.680849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:58.680857 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:58.681144 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:58.681146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:58.681834 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:58.681835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:58.682437 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:58.682450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:58.682587 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:58.684052 (+ 1465us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:08:58.684053 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:58.684082 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:08:58.684105 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:58.684346 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:58.684352 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:58.684355 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:08:58.684429 (+ 74us) client_negotiation.cc:240] Negotiation successful
0131 03:08:58.684525 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":82,"threads_started":1}
I20260131 03:08:58.686338 30040 heartbeater.cc:344] Connected to a master server at 127.19.84.62:38303
I20260131 03:08:58.686452 30040 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:58.686676 30040 heartbeater.cc:507] Master 127.19.84.62:38303 requested a full tablet report, sending...
I20260131 03:08:58.687109 29721 ts_manager.cc:194] Registered new tserver with Master: 29e2df59f1c94c969d4efbb0663efff1 (127.19.84.2:46125)
I20260131 03:08:58.687619 29721 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.19.84.2:47409
W20260131 03:08:58.766521 30043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:58.766732 30043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:58.766765 30043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:58.768232 30043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:58.768307 30043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:58.768348 30043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:08:58.769937 30043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:38303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:58.770219 30043 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:58.770484 30043 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:58.773267 30049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:58.773274 30051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:58.773293 30048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:58.773626 30043 server_base.cc:1060] running on GCE node
I20260131 03:08:58.773805 30043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:58.773984 30043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:58.775159 30043 hybrid_clock.cc:648] HybridClock initialized: now 1769828938775137 us; error 31 us; skew 500 ppm
I20260131 03:08:58.776399 30043 webserver.cc:492] Webserver started at http://127.19.84.3:39859/ using document root <none> and password file <none>
I20260131 03:08:58.776628 30043 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:58.776692 30043 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:58.776834 30043 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:58.777786 30043 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "09a14f368fb34076bfd980bb83fd8c84"
format_stamp: "Formatted at 2026-01-31 03:08:58 on dist-test-slave-5nnp"
server_key: "e4698fe60304c492fbb5bf58c54e2407"
server_key_iv: "f388dd03137527b6ad54ba84632ca8bd"
server_key_version: "encryptionkey@0"
I20260131 03:08:58.778112 30043 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "09a14f368fb34076bfd980bb83fd8c84"
format_stamp: "Formatted at 2026-01-31 03:08:58 on dist-test-slave-5nnp"
server_key: "e4698fe60304c492fbb5bf58c54e2407"
server_key_iv: "f388dd03137527b6ad54ba84632ca8bd"
server_key_version: "encryptionkey@0"
I20260131 03:08:58.779493 30043 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260131 03:08:58.780336 30057 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:58.780534 30043 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:58.780624 30043 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "09a14f368fb34076bfd980bb83fd8c84"
format_stamp: "Formatted at 2026-01-31 03:08:58 on dist-test-slave-5nnp"
server_key: "e4698fe60304c492fbb5bf58c54e2407"
server_key_iv: "f388dd03137527b6ad54ba84632ca8bd"
server_key_version: "encryptionkey@0"
I20260131 03:08:58.780702 30043 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:58.791011 30043 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:58.791332 30043 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:58.791507 30043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:58.791731 30043 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:58.792063 30043 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:58.792119 30043 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:58.792164 30043 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:58.792189 30043 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:58.798460 30043 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:44205
I20260131 03:08:58.798560 30170 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:44205 every 8 connection(s)
I20260131 03:08:58.799028 30043 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
I20260131 03:08:58.799172 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 30043
I20260131 03:08:58.799338 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:08:58.799674 19792 external_mini_cluster.cc:1468] Setting key ce43a5cc292eeeb8d19f9572ef640e2d
I20260131 03:08:58.803633 29760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:58.800002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:60065 (local address 127.19.84.62:38303)
0131 03:08:58.800056 (+ 54us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:58.800059 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:58.800337 (+ 278us) server_negotiation.cc:408] Connection header received
0131 03:08:58.800858 (+ 521us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:58.800859 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:58.800869 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:58.800908 (+ 39us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:58.801373 (+ 465us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:58.801888 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:08:58.802527 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:08:58.802657 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:58.803242 (+ 585us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:58.803247 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:58.803248 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:08:58.803253 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:58.803269 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:08:58.803273 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:58.803336 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:58.803538 (+ 202us) server_negotiation.cc:300] Negotiation successful
0131 03:08:58.803569 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260131 03:08:58.803772 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:58.799979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:38303 (local address 127.19.84.3:60065)
0131 03:08:58.800182 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:58.800215 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:58.800757 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:58.800955 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:58.800959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:58.800968 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:58.801276 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:58.801278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:58.801987 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:08:58.801988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:08:58.802424 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:08:58.802425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:08:58.802546 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:08:58.803082 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:08:58.803083 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:58.803104 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:08:58.803122 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:58.803369 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:58.803423 (+ 54us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:58.803426 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:08:58.803516 (+ 90us) client_negotiation.cc:240] Negotiation successful
0131 03:08:58.803594 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260131 03:08:58.804440 30171 heartbeater.cc:344] Connected to a master server at 127.19.84.62:38303
I20260131 03:08:58.804574 30171 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:58.804806 30171 heartbeater.cc:507] Master 127.19.84.62:38303 requested a full tablet report, sending...
I20260131 03:08:58.805258 29721 ts_manager.cc:194] Registered new tserver with Master: 09a14f368fb34076bfd980bb83fd8c84 (127.19.84.3:44205)
I20260131 03:08:58.805657 29721 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.19.84.3:60065
I20260131 03:08:58.812089 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260131 03:08:58.814061 29760 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:58.813571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56358 (local address 127.19.84.62:38303)
0131 03:08:58.813634 (+ 63us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:58.813637 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:58.813717 (+ 80us) server_negotiation.cc:408] Connection header received
0131 03:08:58.813771 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:58.813772 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:58.813780 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:58.813841 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:58.813972 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56358: BlockingRecv error: recv got EOF from 127.0.0.1:56358 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260131 03:08:58.814182 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 29781
I20260131 03:08:58.818789 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 29912
I20260131 03:08:58.823203 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 30043
I20260131 03:08:58.827270 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 29690
2026-01-31T03:08:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (583 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-01-31T03:08:58Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:08:58Z Disabled control of system clock
I20260131 03:08:58.853736 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:35229
--webserver_interface=127.19.84.62
--webserver_port=0
--builtin_ntp_servers=127.19.84.20:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:35229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260131 03:08:58.933640 30182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:58.933840 30182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:58.933863 30182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:58.935328 30182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:08:58.935400 30182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:58.935413 30182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:58.935475 30182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:08:58.935499 30182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:08:58.937105 30182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:35229
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:35229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:58.937330 30182 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:58.937518 30182 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:58.940222 30190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:58.940277 30188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:58.940253 30187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:58.940553 30182 server_base.cc:1060] running on GCE node
I20260131 03:08:58.940793 30182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:58.941013 30182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:58.942160 30182 hybrid_clock.cc:648] HybridClock initialized: now 1769828938942149 us; error 28 us; skew 500 ppm
I20260131 03:08:58.943574 30182 webserver.cc:492] Webserver started at http://127.19.84.62:36797/ using document root <none> and password file <none>
I20260131 03:08:58.943802 30182 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:58.943866 30182 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:58.943984 30182 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:58.944968 30182 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "b628d75fa0724bbf991457d60fc52f91"
format_stamp: "Formatted at 2026-01-31 03:08:58 on dist-test-slave-5nnp"
server_key: "2b08f287b4d9371836ca74ad52f8a334"
server_key_iv: "d2efee47501aa39602694b54b4293ad8"
server_key_version: "encryptionkey@0"
I20260131 03:08:58.945317 30182 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "b628d75fa0724bbf991457d60fc52f91"
format_stamp: "Formatted at 2026-01-31 03:08:58 on dist-test-slave-5nnp"
server_key: "2b08f287b4d9371836ca74ad52f8a334"
server_key_iv: "d2efee47501aa39602694b54b4293ad8"
server_key_version: "encryptionkey@0"
I20260131 03:08:58.946635 30182 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:08:58.947613 30196 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:58.947858 30182 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:58.947955 30182 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "b628d75fa0724bbf991457d60fc52f91"
format_stamp: "Formatted at 2026-01-31 03:08:58 on dist-test-slave-5nnp"
server_key: "2b08f287b4d9371836ca74ad52f8a334"
server_key_iv: "d2efee47501aa39602694b54b4293ad8"
server_key_version: "encryptionkey@0"
I20260131 03:08:58.948038 30182 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:58.951584 30182 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:58.951846 30182 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:58.951968 30182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:58.955962 30182 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:35229
I20260131 03:08:58.956025 30248 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:35229 every 8 connection(s)
I20260131 03:08:58.956379 30182 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:08:58.956984 30249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:08:58.957876 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 30182
I20260131 03:08:58.957991 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:08:58.958184 19792 external_mini_cluster.cc:1468] Setting key 0122d8ad9ef31d321ce05e8778d2891e
I20260131 03:08:58.960197 30249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b628d75fa0724bbf991457d60fc52f91: Bootstrap starting.
I20260131 03:08:58.960808 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:58.958801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52510 (local address 127.19.84.62:35229)
0131 03:08:58.958999 (+ 198us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:58.959007 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:58.959056 (+ 49us) server_negotiation.cc:408] Connection header received
0131 03:08:58.959504 (+ 448us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:58.959510 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:58.959547 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:58.959681 (+ 134us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:58.960407 (+ 726us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:58.960410 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:58.960410 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:08:58.960415 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:58.960453 (+ 38us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:08:58.960462 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:58.960565 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:58.960605 (+ 40us) server_negotiation.cc:300] Negotiation successful
0131 03:08:58.960653 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260131 03:08:58.960889 30249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b628d75fa0724bbf991457d60fc52f91: Neither blocks nor log segments found. Creating new log.
I20260131 03:08:58.961167 30249 log.cc:826] T 00000000000000000000000000000000 P b628d75fa0724bbf991457d60fc52f91: Log is configured to *not* fsync() on all Append() calls
I20260131 03:08:58.961856 30249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b628d75fa0724bbf991457d60fc52f91: No bootstrap required, opened a new log
I20260131 03:08:58.962945 30249 raft_consensus.cc:359] T 00000000000000000000000000000000 P b628d75fa0724bbf991457d60fc52f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b628d75fa0724bbf991457d60fc52f91" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 35229 } }
I20260131 03:08:58.963094 30249 raft_consensus.cc:385] T 00000000000000000000000000000000 P b628d75fa0724bbf991457d60fc52f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:08:58.963137 30249 raft_consensus.cc:740] T 00000000000000000000000000000000 P b628d75fa0724bbf991457d60fc52f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b628d75fa0724bbf991457d60fc52f91, State: Initialized, Role: FOLLOWER
I20260131 03:08:58.963233 30249 consensus_queue.cc:260] T 00000000000000000000000000000000 P b628d75fa0724bbf991457d60fc52f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b628d75fa0724bbf991457d60fc52f91" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 35229 } }
I20260131 03:08:58.963325 30249 raft_consensus.cc:399] T 00000000000000000000000000000000 P b628d75fa0724bbf991457d60fc52f91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:08:58.963366 30249 raft_consensus.cc:493] T 00000000000000000000000000000000 P b628d75fa0724bbf991457d60fc52f91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:08:58.963439 30249 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b628d75fa0724bbf991457d60fc52f91 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:08:58.964000 30249 raft_consensus.cc:515] T 00000000000000000000000000000000 P b628d75fa0724bbf991457d60fc52f91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b628d75fa0724bbf991457d60fc52f91" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 35229 } }
I20260131 03:08:58.964129 30249 leader_election.cc:304] T 00000000000000000000000000000000 P b628d75fa0724bbf991457d60fc52f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b628d75fa0724bbf991457d60fc52f91; no voters:
I20260131 03:08:58.964282 30249 leader_election.cc:290] T 00000000000000000000000000000000 P b628d75fa0724bbf991457d60fc52f91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:08:58.964339 30254 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b628d75fa0724bbf991457d60fc52f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:08:58.964504 30254 raft_consensus.cc:697] T 00000000000000000000000000000000 P b628d75fa0724bbf991457d60fc52f91 [term 1 LEADER]: Becoming Leader. State: Replica: b628d75fa0724bbf991457d60fc52f91, State: Running, Role: LEADER
I20260131 03:08:58.964540 30249 sys_catalog.cc:565] T 00000000000000000000000000000000 P b628d75fa0724bbf991457d60fc52f91 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:08:58.964612 30254 consensus_queue.cc:237] T 00000000000000000000000000000000 P b628d75fa0724bbf991457d60fc52f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b628d75fa0724bbf991457d60fc52f91" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 35229 } }
I20260131 03:08:58.965032 30256 sys_catalog.cc:455] T 00000000000000000000000000000000 P b628d75fa0724bbf991457d60fc52f91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b628d75fa0724bbf991457d60fc52f91. Latest consensus state: current_term: 1 leader_uuid: "b628d75fa0724bbf991457d60fc52f91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b628d75fa0724bbf991457d60fc52f91" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 35229 } } }
I20260131 03:08:58.965018 30255 sys_catalog.cc:455] T 00000000000000000000000000000000 P b628d75fa0724bbf991457d60fc52f91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b628d75fa0724bbf991457d60fc52f91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b628d75fa0724bbf991457d60fc52f91" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 35229 } } }
I20260131 03:08:58.965178 30256 sys_catalog.cc:458] T 00000000000000000000000000000000 P b628d75fa0724bbf991457d60fc52f91 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:58.965185 30255 sys_catalog.cc:458] T 00000000000000000000000000000000 P b628d75fa0724bbf991457d60fc52f91 [sys.catalog]: This master's current role is: LEADER
I20260131 03:08:58.965574 30263 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:08:58.966236 30263 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:08:58.967696 30263 catalog_manager.cc:1357] Generated new cluster ID: 8f4b6db438014465a60a8c80dc28532d
I20260131 03:08:58.967748 30263 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:08:58.986413 30263 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:08:58.986627 30263 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:08:58.995606 30263 catalog_manager.cc:6027] T 00000000000000000000000000000000 P b628d75fa0724bbf991457d60fc52f91: Generated new TSK 0
I20260131 03:08:58.995837 30263 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:08:58.998602 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:35229
--builtin_ntp_servers=127.19.84.20:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260131 03:08:59.085570 30273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:59.085757 30273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:59.085778 30273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:59.087281 30273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:59.087333 30273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:59.087363 30273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:08:59.088985 30273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:35229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:59.089226 30273 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:59.089499 30273 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:59.092247 30278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:59.092383 30273 server_base.cc:1060] running on GCE node
W20260131 03:08:59.092267 30281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:59.092267 30279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:59.092700 30273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:59.092918 30273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:59.094081 30273 hybrid_clock.cc:648] HybridClock initialized: now 1769828939094067 us; error 42 us; skew 500 ppm
I20260131 03:08:59.095345 30273 webserver.cc:492] Webserver started at http://127.19.84.1:41171/ using document root <none> and password file <none>
I20260131 03:08:59.095597 30273 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:59.095667 30273 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:59.095813 30273 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:59.096818 30273 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "99d13e9f89d14f0b9ca267ac0456ed46"
format_stamp: "Formatted at 2026-01-31 03:08:59 on dist-test-slave-5nnp"
server_key: "5595cf377bf6c8cd50d6b76137993f53"
server_key_iv: "a2761c74b2ef4172d7d015d09def2826"
server_key_version: "encryptionkey@0"
I20260131 03:08:59.097154 30273 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "99d13e9f89d14f0b9ca267ac0456ed46"
format_stamp: "Formatted at 2026-01-31 03:08:59 on dist-test-slave-5nnp"
server_key: "5595cf377bf6c8cd50d6b76137993f53"
server_key_iv: "a2761c74b2ef4172d7d015d09def2826"
server_key_version: "encryptionkey@0"
I20260131 03:08:59.098439 30273 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:08:59.099254 30287 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:59.099462 30273 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:59.099573 30273 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "99d13e9f89d14f0b9ca267ac0456ed46"
format_stamp: "Formatted at 2026-01-31 03:08:59 on dist-test-slave-5nnp"
server_key: "5595cf377bf6c8cd50d6b76137993f53"
server_key_iv: "a2761c74b2ef4172d7d015d09def2826"
server_key_version: "encryptionkey@0"
I20260131 03:08:59.099649 30273 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:59.103025 30273 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:59.103274 30273 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:59.103446 30273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:59.103659 30273 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:59.103965 30273 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:59.104019 30273 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:59.104061 30273 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:59.104085 30273 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:59.109544 30273 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:33357
I20260131 03:08:59.109663 30400 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:33357 every 8 connection(s)
I20260131 03:08:59.109934 30273 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
I20260131 03:08:59.112838 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:59.110731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:58785 (local address 127.19.84.62:35229)
0131 03:08:59.110786 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:59.110788 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:59.110986 (+ 198us) server_negotiation.cc:408] Connection header received
0131 03:08:59.111523 (+ 537us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:59.111524 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:59.111531 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:59.111620 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:59.112507 (+ 887us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:59.112509 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:59.112509 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:08:59.112514 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:59.112555 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:08:59.112559 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:59.112627 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:59.112739 (+ 112us) server_negotiation.cc:300] Negotiation successful
0131 03:08:59.112756 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260131 03:08:59.113027 30403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:59.110689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:35229 (local address 127.19.84.1:58785)
0131 03:08:59.110864 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:59.110893 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:59.111362 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:59.111674 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:59.111678 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:59.111686 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:59.112349 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:08:59.112349 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:59.112370 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:08:59.112389 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:59.112626 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:59.112632 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:59.112635 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:08:59.112768 (+ 133us) client_negotiation.cc:240] Negotiation successful
0131 03:08:59.112818 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
I20260131 03:08:59.113663 30401 heartbeater.cc:344] Connected to a master server at 127.19.84.62:35229
I20260131 03:08:59.113775 30401 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:59.113915 30401 heartbeater.cc:507] Master 127.19.84.62:35229 requested a full tablet report, sending...
I20260131 03:08:59.114385 30213 ts_manager.cc:194] Registered new tserver with Master: 99d13e9f89d14f0b9ca267ac0456ed46 (127.19.84.1:33357)
I20260131 03:08:59.116451 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 30273
I20260131 03:08:59.116582 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:08:59.116778 19792 external_mini_cluster.cc:1468] Setting key 7fbfe51d51dce2e77afc9d4b1db31579
I20260131 03:08:59.117800 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:35229
--builtin_ntp_servers=127.19.84.20:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260131 03:08:59.197346 30404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:59.197556 30404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:59.197593 30404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:59.199023 30404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:59.199090 30404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:59.199132 30404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:08:59.200712 30404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:35229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:59.200956 30404 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:59.201192 30404 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:59.203857 30412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:59.203823 30410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:59.203823 30409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:59.204403 30404 server_base.cc:1060] running on GCE node
I20260131 03:08:59.204605 30404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:59.204818 30404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:59.205948 30404 hybrid_clock.cc:648] HybridClock initialized: now 1769828939205915 us; error 49 us; skew 500 ppm
I20260131 03:08:59.207069 30404 webserver.cc:492] Webserver started at http://127.19.84.2:43851/ using document root <none> and password file <none>
I20260131 03:08:59.207307 30404 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:59.207350 30404 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:59.207508 30404 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:59.208458 30404 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "09be0767a5e44423955e91347aad4ba9"
format_stamp: "Formatted at 2026-01-31 03:08:59 on dist-test-slave-5nnp"
server_key: "f83846a508136a1ac95b5a4795911b34"
server_key_iv: "5c495eb776e56eeb454dc5e47734b2c9"
server_key_version: "encryptionkey@0"
I20260131 03:08:59.208781 30404 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "09be0767a5e44423955e91347aad4ba9"
format_stamp: "Formatted at 2026-01-31 03:08:59 on dist-test-slave-5nnp"
server_key: "f83846a508136a1ac95b5a4795911b34"
server_key_iv: "5c495eb776e56eeb454dc5e47734b2c9"
server_key_version: "encryptionkey@0"
I20260131 03:08:59.210000 30404 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:08:59.210819 30418 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:59.211030 30404 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:59.211102 30404 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "09be0767a5e44423955e91347aad4ba9"
format_stamp: "Formatted at 2026-01-31 03:08:59 on dist-test-slave-5nnp"
server_key: "f83846a508136a1ac95b5a4795911b34"
server_key_iv: "5c495eb776e56eeb454dc5e47734b2c9"
server_key_version: "encryptionkey@0"
I20260131 03:08:59.211153 30404 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:59.214629 30404 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:59.214813 30404 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:59.214895 30404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:59.215040 30404 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:59.215289 30404 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:59.215322 30404 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:59.215341 30404 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:59.215354 30404 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:59.221115 30404 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:39567
I20260131 03:08:59.221184 30531 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:39567 every 8 connection(s)
I20260131 03:08:59.221573 30404 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
I20260131 03:08:59.222218 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 30404
I20260131 03:08:59.222371 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:08:59.222566 19792 external_mini_cluster.cc:1468] Setting key d2126c8f22394030e371706dbfbb311e
I20260131 03:08:59.223671 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:35229
--builtin_ntp_servers=127.19.84.20:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260131 03:08:59.224418 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:59.222351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:49097 (local address 127.19.84.62:35229)
0131 03:08:59.222402 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:59.222404 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:59.222688 (+ 284us) server_negotiation.cc:408] Connection header received
0131 03:08:59.223254 (+ 566us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:59.223256 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:59.223267 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:59.223319 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:59.224114 (+ 795us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:59.224115 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:59.224116 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:08:59.224121 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:59.224138 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:08:59.224142 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:59.224206 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:59.224289 (+ 83us) server_negotiation.cc:300] Negotiation successful
0131 03:08:59.224306 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260131 03:08:59.224507 30534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:59.222328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:35229 (local address 127.19.84.2:49097)
0131 03:08:59.222580 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:59.222605 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:59.223141 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:59.223406 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:59.223410 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:59.223422 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:59.223984 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:08:59.223984 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:59.224004 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:08:59.224023 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:59.224211 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:59.224215 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:59.224218 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:08:59.224292 (+ 74us) client_negotiation.cc:240] Negotiation successful
0131 03:08:59.224333 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":75,"threads_started":1}
I20260131 03:08:59.225020 30532 heartbeater.cc:344] Connected to a master server at 127.19.84.62:35229
I20260131 03:08:59.225145 30532 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:59.225384 30532 heartbeater.cc:507] Master 127.19.84.62:35229 requested a full tablet report, sending...
I20260131 03:08:59.225775 30213 ts_manager.cc:194] Registered new tserver with Master: 09be0767a5e44423955e91347aad4ba9 (127.19.84.2:39567)
W20260131 03:08:59.302165 30535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:08:59.302333 30535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:08:59.302356 30535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:08:59.303848 30535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:08:59.303886 30535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:08:59.303911 30535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:08:59.305359 30535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:35229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:08:59.305518 30535 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:08:59.305694 30535 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:08:59.308354 30543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:59.308378 30540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:08:59.308457 30541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:08:59.308730 30535 server_base.cc:1060] running on GCE node
I20260131 03:08:59.308897 30535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:08:59.309095 30535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:08:59.310266 30535 hybrid_clock.cc:648] HybridClock initialized: now 1769828939310251 us; error 22 us; skew 500 ppm
I20260131 03:08:59.311414 30535 webserver.cc:492] Webserver started at http://127.19.84.3:39113/ using document root <none> and password file <none>
I20260131 03:08:59.311628 30535 fs_manager.cc:362] Metadata directory not provided
I20260131 03:08:59.311686 30535 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:08:59.311805 30535 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:08:59.312754 30535 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "89823cf7e88f425c91be9ec38b445726"
format_stamp: "Formatted at 2026-01-31 03:08:59 on dist-test-slave-5nnp"
server_key: "56e02de92c52ec988e79e5b33740a143"
server_key_iv: "2453c39d4fb55a754a1c8381ccc91713"
server_key_version: "encryptionkey@0"
I20260131 03:08:59.313072 30535 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "89823cf7e88f425c91be9ec38b445726"
format_stamp: "Formatted at 2026-01-31 03:08:59 on dist-test-slave-5nnp"
server_key: "56e02de92c52ec988e79e5b33740a143"
server_key_iv: "2453c39d4fb55a754a1c8381ccc91713"
server_key_version: "encryptionkey@0"
I20260131 03:08:59.314275 30535 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:08:59.315057 30549 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:59.315268 30535 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:08:59.315354 30535 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "89823cf7e88f425c91be9ec38b445726"
format_stamp: "Formatted at 2026-01-31 03:08:59 on dist-test-slave-5nnp"
server_key: "56e02de92c52ec988e79e5b33740a143"
server_key_iv: "2453c39d4fb55a754a1c8381ccc91713"
server_key_version: "encryptionkey@0"
I20260131 03:08:59.315445 30535 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:08:59.318727 30535 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:08:59.318928 30535 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:08:59.319047 30535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:08:59.319231 30535 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:08:59.319564 30535 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:08:59.319615 30535 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:59.319657 30535 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:08:59.319681 30535 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:08:59.325263 30535 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:35237
I20260131 03:08:59.325338 30662 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:35237 every 8 connection(s)
I20260131 03:08:59.325636 30535 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
I20260131 03:08:59.328027 30252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:59.326375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:46583 (local address 127.19.84.62:35229)
0131 03:08:59.326439 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:59.326441 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:59.326584 (+ 143us) server_negotiation.cc:408] Connection header received
0131 03:08:59.326997 (+ 413us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:59.326998 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:59.327003 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:59.327064 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:59.327763 (+ 699us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:08:59.327765 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:08:59.327765 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:08:59.327769 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:08:59.327784 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:08:59.327787 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:08:59.327837 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0131 03:08:59.327963 (+ 126us) server_negotiation.cc:300] Negotiation successful
0131 03:08:59.327975 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260131 03:08:59.328222 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:08:59.326296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:35229 (local address 127.19.84.3:46583)
0131 03:08:59.326466 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0131 03:08:59.326503 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0131 03:08:59.326929 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:08:59.327126 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:08:59.327129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:08:59.327137 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:08:59.327649 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:08:59.327650 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:08:59.327668 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:08:59.327685 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:08:59.327877 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:08:59.327882 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:08:59.327885 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:08:59.327985 (+ 100us) client_negotiation.cc:240] Negotiation successful
0131 03:08:59.328028 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
I20260131 03:08:59.328436 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 30535
I20260131 03:08:59.328542 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:08:59.328663 30663 heartbeater.cc:344] Connected to a master server at 127.19.84.62:35229
I20260131 03:08:59.328742 19792 external_mini_cluster.cc:1468] Setting key 7cca07c30678c6b2a453cf991d6a8b69
I20260131 03:08:59.328755 30663 heartbeater.cc:461] Registering TS with master...
I20260131 03:08:59.328876 30663 heartbeater.cc:507] Master 127.19.84.62:35229 requested a full tablet report, sending...
I20260131 03:08:59.329253 30213 ts_manager.cc:194] Registered new tserver with Master: 89823cf7e88f425c91be9ec38b445726 (127.19.84.3:35237)
I20260131 03:08:59.329504 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260131 03:08:59.331411 30252 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:08:59.330873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52526 (local address 127.19.84.62:35229)
0131 03:08:59.330938 (+ 65us) server_negotiation.cc:207] Beginning negotiation
0131 03:08:59.330940 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:08:59.331028 (+ 88us) server_negotiation.cc:408] Connection header received
0131 03:08:59.331109 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:08:59.331110 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:08:59.331115 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:08:59.331190 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:08:59.331311 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52526: BlockingRecv error: recv got EOF from 127.0.0.1:52526 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260131 03:08:59.331521 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 30273
I20260131 03:08:59.337453 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 30404
I20260131 03:08:59.342278 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 30535
I20260131 03:08:59.346606 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 30182
2026-01-31T03:08:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (518 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:08:59 dist-test-slave-5nnp krb5kdc[30673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:08:59 dist-test-slave-5nnp krb5kdc[30673](info): set up 2 sockets
Jan 31 03:08:59 dist-test-slave-5nnp krb5kdc[30673](info): commencing operation
krb5kdc: starting...
W20260131 03:09:01.346547 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.980s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:09:01 dist-test-slave-5nnp krb5kdc[30673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:09:01Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:09:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:09:01.499233 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:45169
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:41049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:45169
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:09:01.576874 30689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:09:01.577086 30689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:09:01.577121 30689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:09:01.578544 30689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:09:01.578617 30689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:09:01.578642 30689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:09:01.578670 30689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:09:01.578697 30689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:09:01.580340 30689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:41049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:45169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:45169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:09:01.580590 30689 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:09:01.580785 30689 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:09:01.583547 30697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:09:01.583559 30695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:09:01.583551 30694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:09:01.583575 30689 server_base.cc:1060] running on GCE node
I20260131 03:09:01.584087 30689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:09:01.584364 30689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:09:01.585513 30689 hybrid_clock.cc:648] HybridClock initialized: now 1769828941585509 us; error 53 us; skew 500 ppm
Jan 31 03:09:01 dist-test-slave-5nnp krb5kdc[30673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828941, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:09:01.587941 30689 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:09:01.588508 30689 webserver.cc:492] Webserver started at http://127.19.84.62:44713/ using document root <none> and password file <none>
I20260131 03:09:01.588709 30689 fs_manager.cc:362] Metadata directory not provided
I20260131 03:09:01.588763 30689 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:09:01.588904 30689 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:09:01.589893 30689 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "ef8a283c14254471919ad837154b9609"
format_stamp: "Formatted at 2026-01-31 03:09:01 on dist-test-slave-5nnp"
server_key: "0ca281915b34c4e91b414481a0c3c701"
server_key_iv: "1ad63966867f0c9c6e43bacb4f52f6cb"
server_key_version: "encryptionkey@0"
I20260131 03:09:01.590212 30689 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "ef8a283c14254471919ad837154b9609"
format_stamp: "Formatted at 2026-01-31 03:09:01 on dist-test-slave-5nnp"
server_key: "0ca281915b34c4e91b414481a0c3c701"
server_key_iv: "1ad63966867f0c9c6e43bacb4f52f6cb"
server_key_version: "encryptionkey@0"
I20260131 03:09:01.591491 30689 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:09:01.592248 30704 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:09:01.592429 30689 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:09:01.592522 30689 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "ef8a283c14254471919ad837154b9609"
format_stamp: "Formatted at 2026-01-31 03:09:01 on dist-test-slave-5nnp"
server_key: "0ca281915b34c4e91b414481a0c3c701"
server_key_iv: "1ad63966867f0c9c6e43bacb4f52f6cb"
server_key_version: "encryptionkey@0"
I20260131 03:09:01.592602 30689 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:09:01.614629 30689 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:09:01.616701 30689 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:09:01.616864 30689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:09:01.621085 30689 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:45169
I20260131 03:09:01.621117 30756 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:45169 every 8 connection(s)
I20260131 03:09:01.621450 30689 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:09:01.622079 30757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:09:01.623700 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 30689
I20260131 03:09:01.623822 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:09:01.623996 19792 external_mini_cluster.cc:1468] Setting key 2688abbb711eeec3316b6eab8ae9ed2b
I20260131 03:09:01.624990 30757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef8a283c14254471919ad837154b9609: Bootstrap starting.
I20260131 03:09:01.625576 30757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef8a283c14254471919ad837154b9609: Neither blocks nor log segments found. Creating new log.
I20260131 03:09:01.625828 30757 log.cc:826] T 00000000000000000000000000000000 P ef8a283c14254471919ad837154b9609: Log is configured to *not* fsync() on all Append() calls
I20260131 03:09:01.626479 30757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef8a283c14254471919ad837154b9609: No bootstrap required, opened a new log
I20260131 03:09:01.627588 30757 raft_consensus.cc:359] T 00000000000000000000000000000000 P ef8a283c14254471919ad837154b9609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8a283c14254471919ad837154b9609" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 45169 } }
I20260131 03:09:01.627723 30757 raft_consensus.cc:385] T 00000000000000000000000000000000 P ef8a283c14254471919ad837154b9609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:09:01.627760 30757 raft_consensus.cc:740] T 00000000000000000000000000000000 P ef8a283c14254471919ad837154b9609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef8a283c14254471919ad837154b9609, State: Initialized, Role: FOLLOWER
I20260131 03:09:01.627856 30757 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef8a283c14254471919ad837154b9609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8a283c14254471919ad837154b9609" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 45169 } }
I20260131 03:09:01.627949 30757 raft_consensus.cc:399] T 00000000000000000000000000000000 P ef8a283c14254471919ad837154b9609 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:09:01.628000 30757 raft_consensus.cc:493] T 00000000000000000000000000000000 P ef8a283c14254471919ad837154b9609 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:09:01.628049 30757 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ef8a283c14254471919ad837154b9609 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:09:01.628701 30757 raft_consensus.cc:515] T 00000000000000000000000000000000 P ef8a283c14254471919ad837154b9609 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8a283c14254471919ad837154b9609" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 45169 } }
I20260131 03:09:01.628832 30757 leader_election.cc:304] T 00000000000000000000000000000000 P ef8a283c14254471919ad837154b9609 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef8a283c14254471919ad837154b9609; no voters:
I20260131 03:09:01.628981 30757 leader_election.cc:290] T 00000000000000000000000000000000 P ef8a283c14254471919ad837154b9609 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:09:01.629056 30762 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ef8a283c14254471919ad837154b9609 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:09:01.629197 30762 raft_consensus.cc:697] T 00000000000000000000000000000000 P ef8a283c14254471919ad837154b9609 [term 1 LEADER]: Becoming Leader. State: Replica: ef8a283c14254471919ad837154b9609, State: Running, Role: LEADER
I20260131 03:09:01.629235 30757 sys_catalog.cc:565] T 00000000000000000000000000000000 P ef8a283c14254471919ad837154b9609 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:09:01.629298 30762 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef8a283c14254471919ad837154b9609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8a283c14254471919ad837154b9609" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 45169 } }
I20260131 03:09:01.629626 30763 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef8a283c14254471919ad837154b9609 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef8a283c14254471919ad837154b9609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8a283c14254471919ad837154b9609" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 45169 } } }
I20260131 03:09:01.629849 30763 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef8a283c14254471919ad837154b9609 [sys.catalog]: This master's current role is: LEADER
Jan 31 03:09:01 dist-test-slave-5nnp krb5kdc[30673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:09:01.629618 30762 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef8a283c14254471919ad837154b9609 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef8a283c14254471919ad837154b9609. Latest consensus state: current_term: 1 leader_uuid: "ef8a283c14254471919ad837154b9609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8a283c14254471919ad837154b9609" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 45169 } } }
I20260131 03:09:01.630013 30762 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef8a283c14254471919ad837154b9609 [sys.catalog]: This master's current role is: LEADER
I20260131 03:09:01.630129 30770 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:09:01.630616 30770 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:09:01.633280 30770 catalog_manager.cc:1357] Generated new cluster ID: 8d69d137a5514d2f863f78b1c70ee44d
I20260131 03:09:01.633337 30770 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:09:01.634138 30760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:09:01.624549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45174 (local address 127.19.84.62:45169)
0131 03:09:01.624757 (+ 208us) server_negotiation.cc:207] Beginning negotiation
0131 03:09:01.624769 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0131 03:09:01.624799 (+ 30us) server_negotiation.cc:408] Connection header received
0131 03:09:01.625167 (+ 368us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:09:01.625173 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:09:01.625216 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:09:01.625351 (+ 135us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:09:01.626205 (+ 854us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:09:01.627013 (+ 808us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:09:01.627658 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:09:01.627946 (+ 288us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:09:01.630583 (+ 2637us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:09:01.630590 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:09:01.630591 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:09:01.630613 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:09:01.632647 (+ 2034us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:09:01.633046 (+ 399us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:09:01.633049 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:09:01.633049 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:09:01.633114 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:09:01.633300 (+ 186us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:09:01.633301 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:09:01.633302 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:09:01.633553 (+ 251us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:09:01.633701 (+ 148us) server_negotiation.cc:1036] Waiting for connection context
0131 03:09:01.633825 (+ 124us) server_negotiation.cc:300] Negotiation successful
0131 03:09:01.633949 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":88,"threads_started":1}
I20260131 03:09:01.643903 30770 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:09:01.644479 30770 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:09:01.647919 30770 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ef8a283c14254471919ad837154b9609: Generated new TSK 0
I20260131 03:09:01.648119 30770 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:09:01.710919 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:45169
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:41049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:09:01.795130 30785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:09:01.795320 30785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:09:01.795346 30785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:09:01.797010 30785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:09:01.797061 30785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:09:01.797089 30785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:09:01.798775 30785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:41049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:45169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:09:01.799023 30785 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:09:01.799252 30785 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:09:01.802170 30791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:09:01.802196 30793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:09:01.802309 30785 server_base.cc:1060] running on GCE node
W20260131 03:09:01.802196 30790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:09:01.802624 30785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:09:01.802870 30785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:09:01.804025 30785 hybrid_clock.cc:648] HybridClock initialized: now 1769828941803982 us; error 68 us; skew 500 ppm
Jan 31 03:09:01 dist-test-slave-5nnp krb5kdc[30673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828941, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:09:01.806486 30785 init.cc:377] Logged in from keytab as kudu/127.19.84.1@KRBTEST.COM (short username kudu)
I20260131 03:09:01.807032 30785 webserver.cc:492] Webserver started at http://127.19.84.1:43929/ using document root <none> and password file <none>
I20260131 03:09:01.807237 30785 fs_manager.cc:362] Metadata directory not provided
I20260131 03:09:01.807294 30785 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:09:01.807471 30785 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:09:01.808454 30785 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "838c2ab5e1d14bcd9435831f92d269c9"
format_stamp: "Formatted at 2026-01-31 03:09:01 on dist-test-slave-5nnp"
server_key: "fda1d94c0f79c195309ec73445fd138a"
server_key_iv: "726fdf0372266060677cdb22c1da0b72"
server_key_version: "encryptionkey@0"
I20260131 03:09:01.808784 30785 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "838c2ab5e1d14bcd9435831f92d269c9"
format_stamp: "Formatted at 2026-01-31 03:09:01 on dist-test-slave-5nnp"
server_key: "fda1d94c0f79c195309ec73445fd138a"
server_key_iv: "726fdf0372266060677cdb22c1da0b72"
server_key_version: "encryptionkey@0"
I20260131 03:09:01.810045 30785 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260131 03:09:01.810860 30800 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:09:01.811084 30785 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:09:01.811223 30785 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "838c2ab5e1d14bcd9435831f92d269c9"
format_stamp: "Formatted at 2026-01-31 03:09:01 on dist-test-slave-5nnp"
server_key: "fda1d94c0f79c195309ec73445fd138a"
server_key_iv: "726fdf0372266060677cdb22c1da0b72"
server_key_version: "encryptionkey@0"
I20260131 03:09:01.811302 30785 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:09:01.837098 30785 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:09:01.839074 30785 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:09:01.839286 30785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:09:01.839593 30785 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:09:01.839954 30785 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:09:01.840013 30785 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:09:01.840054 30785 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:09:01.840083 30785 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:09:01.846107 30785 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:38137
I20260131 03:09:01.846191 30913 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:38137 every 8 connection(s)
I20260131 03:09:01.846572 30785 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
Jan 31 03:09:01 dist-test-slave-5nnp krb5kdc[30673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828941, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:09:01.855288 30760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:09:01.847401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:51987 (local address 127.19.84.62:45169)
0131 03:09:01.847480 (+ 79us) server_negotiation.cc:207] Beginning negotiation
0131 03:09:01.847482 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:09:01.847665 (+ 183us) server_negotiation.cc:408] Connection header received
0131 03:09:01.848200 (+ 535us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:09:01.848201 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:09:01.848214 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:09:01.848269 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:09:01.849059 (+ 790us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:09:01.849570 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:09:01.850137 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:09:01.850305 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:09:01.852756 (+ 2451us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:09:01.852761 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:09:01.852762 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:09:01.852770 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:09:01.854003 (+ 1233us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:09:01.854396 (+ 393us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:09:01.854398 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:09:01.854398 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:09:01.854428 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:09:01.854653 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:09:01.854654 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:09:01.854655 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:09:01.854755 (+ 100us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:09:01.854831 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0131 03:09:01.855095 (+ 264us) server_negotiation.cc:300] Negotiation successful
0131 03:09:01.855182 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260131 03:09:01.855465 30916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:09:01.847290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:45169 (local address 127.19.84.1:51987)
0131 03:09:01.847517 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0131 03:09:01.847583 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0131 03:09:01.848102 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:09:01.848341 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:09:01.848345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:09:01.848661 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:09:01.848950 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:09:01.848954 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:09:01.849665 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:09:01.849667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:09:01.850051 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:09:01.850052 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:09:01.850206 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:09:01.851162 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:09:01.851180 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:09:01.852657 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:09:01.854094 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:09:01.854100 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:09:01.854104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:09:01.854318 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:09:01.854493 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:09:01.854494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:09:01.854494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:09:01.854585 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:09:01.854822 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:09:01.854827 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:09:01.854959 (+ 132us) client_negotiation.cc:769] Sending connection context
0131 03:09:01.855076 (+ 117us) client_negotiation.cc:240] Negotiation successful
0131 03:09:01.855217 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":111,"threads_started":1}
I20260131 03:09:01.855913 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 30785
I20260131 03:09:01.856030 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:09:01.856019 30914 heartbeater.cc:344] Connected to a master server at 127.19.84.62:45169
I20260131 03:09:01.856120 30914 heartbeater.cc:461] Registering TS with master...
I20260131 03:09:01.856243 19792 external_mini_cluster.cc:1468] Setting key d78bf3662553ebbf1ab4ed1e6fd739a0
I20260131 03:09:01.856377 30914 heartbeater.cc:507] Master 127.19.84.62:45169 requested a full tablet report, sending...
I20260131 03:09:01.856877 30720 ts_manager.cc:194] Registered new tserver with Master: 838c2ab5e1d14bcd9435831f92d269c9 (127.19.84.1:38137)
I20260131 03:09:01.857911 30720 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.1@KRBTEST.COM'} at 127.19.84.1:51987
WARNING: no policy specified for kudu/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:09:01.904191 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:45169
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:41049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:09:01.982832 30921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:09:01.982996 30921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:09:01.983019 30921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:09:01.984522 30921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:09:01.984573 30921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:09:01.984601 30921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:09:01.986358 30921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:41049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:45169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:09:01.986585 30921 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:09:01.986735 30921 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:09:01.989840 30926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:09:01.989903 30927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:09:01.990274 30929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:09:01.990394 30921 server_base.cc:1060] running on GCE node
I20260131 03:09:01.990612 30921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:09:01.990852 30921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:09:01.992008 30921 hybrid_clock.cc:648] HybridClock initialized: now 1769828941991988 us; error 27 us; skew 500 ppm
Jan 31 03:09:01 dist-test-slave-5nnp krb5kdc[30673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828941, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:09:01.994895 30921 init.cc:377] Logged in from keytab as kudu/127.19.84.2@KRBTEST.COM (short username kudu)
I20260131 03:09:01.995546 30921 webserver.cc:492] Webserver started at http://127.19.84.2:45133/ using document root <none> and password file <none>
I20260131 03:09:01.995774 30921 fs_manager.cc:362] Metadata directory not provided
I20260131 03:09:01.995811 30921 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:09:01.995896 30921 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:09:01.997004 30921 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "68cde00d385549c499d650b62353e539"
format_stamp: "Formatted at 2026-01-31 03:09:01 on dist-test-slave-5nnp"
server_key: "4430b521ff05679373260f7b34a514b1"
server_key_iv: "02b675d3b021f09a5ef052f0c84909cf"
server_key_version: "encryptionkey@0"
I20260131 03:09:01.997395 30921 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "68cde00d385549c499d650b62353e539"
format_stamp: "Formatted at 2026-01-31 03:09:01 on dist-test-slave-5nnp"
server_key: "4430b521ff05679373260f7b34a514b1"
server_key_iv: "02b675d3b021f09a5ef052f0c84909cf"
server_key_version: "encryptionkey@0"
I20260131 03:09:01.998679 30921 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:09:01.999349 30936 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:09:01.999555 30921 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260131 03:09:01.999641 30921 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "68cde00d385549c499d650b62353e539"
format_stamp: "Formatted at 2026-01-31 03:09:01 on dist-test-slave-5nnp"
server_key: "4430b521ff05679373260f7b34a514b1"
server_key_iv: "02b675d3b021f09a5ef052f0c84909cf"
server_key_version: "encryptionkey@0"
I20260131 03:09:01.999719 30921 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:09:02.009116 30921 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:09:02.011154 30921 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:09:02.011324 30921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:09:02.011598 30921 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:09:02.011921 30921 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:09:02.011983 30921 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:09:02.012027 30921 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:09:02.012198 30921 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:09:02.019068 30921 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:36389
I20260131 03:09:02.019173 31049 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:36389 every 8 connection(s)
I20260131 03:09:02.019553 30921 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
Jan 31 03:09:02 dist-test-slave-5nnp krb5kdc[30673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828941, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:09:02.028463 30760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:09:02.020334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:57765 (local address 127.19.84.62:45169)
0131 03:09:02.020402 (+ 68us) server_negotiation.cc:207] Beginning negotiation
0131 03:09:02.020404 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:09:02.020558 (+ 154us) server_negotiation.cc:408] Connection header received
0131 03:09:02.021026 (+ 468us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:09:02.021027 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:09:02.021036 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:09:02.021101 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:09:02.021899 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.022403 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:09:02.022952 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.023070 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:09:02.025470 (+ 2400us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:09:02.025482 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:09:02.025482 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:09:02.025489 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:09:02.026996 (+ 1507us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:09:02.027503 (+ 507us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:09:02.027505 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:09:02.027505 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:09:02.027539 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:09:02.027773 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:09:02.027774 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:09:02.027774 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:09:02.027884 (+ 110us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:09:02.027962 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0131 03:09:02.028230 (+ 268us) server_negotiation.cc:300] Negotiation successful
0131 03:09:02.028357 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260131 03:09:02.028635 31052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:09:02.020269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:45169 (local address 127.19.84.2:57765)
0131 03:09:02.020429 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0131 03:09:02.020463 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0131 03:09:02.020946 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:09:02.021165 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:09:02.021169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:09:02.021482 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:09:02.021788 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:09:02.021791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.022499 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:09:02.022500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:09:02.022880 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:09:02.022881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.022990 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:09:02.023939 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:09:02.023956 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:09:02.025360 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:09:02.027124 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:09:02.027131 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:09:02.027134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:09:02.027349 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:09:02.027613 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:09:02.027614 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:09:02.027614 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:09:02.027705 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:09:02.027951 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:09:02.027956 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:09:02.028094 (+ 138us) client_negotiation.cc:769] Sending connection context
0131 03:09:02.028212 (+ 118us) client_negotiation.cc:240] Negotiation successful
0131 03:09:02.028356 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260131 03:09:02.028868 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 30921
I20260131 03:09:02.028983 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:09:02.029198 19792 external_mini_cluster.cc:1468] Setting key 6e1a9f0bd52f4db9590c25511e8f3e9b
I20260131 03:09:02.030229 31050 heartbeater.cc:344] Connected to a master server at 127.19.84.62:45169
I20260131 03:09:02.030359 31050 heartbeater.cc:461] Registering TS with master...
I20260131 03:09:02.030598 31050 heartbeater.cc:507] Master 127.19.84.62:45169 requested a full tablet report, sending...
I20260131 03:09:02.031050 30720 ts_manager.cc:194] Registered new tserver with Master: 68cde00d385549c499d650b62353e539 (127.19.84.2:36389)
I20260131 03:09:02.031610 30720 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.2@KRBTEST.COM'} at 127.19.84.2:57765
WARNING: no policy specified for kudu/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:09:02.078505 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:45169
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:41049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:09:02.159265 31057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:09:02.159622 31057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:09:02.159693 31057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:09:02.161245 31057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:09:02.161316 31057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:09:02.161360 31057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:09:02.162959 31057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:41049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:45169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:09:02.163204 31057 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:09:02.163450 31057 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:09:02.166164 31065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:09:02.166127 31062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:09:02.166249 31063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:09:02.166335 31057 server_base.cc:1060] running on GCE node
I20260131 03:09:02.166555 31057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:09:02.166769 31057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:09:02.167923 31057 hybrid_clock.cc:648] HybridClock initialized: now 1769828942167903 us; error 37 us; skew 500 ppm
Jan 31 03:09:02 dist-test-slave-5nnp krb5kdc[30673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828942, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:09:02.170154 31057 init.cc:377] Logged in from keytab as kudu/127.19.84.3@KRBTEST.COM (short username kudu)
I20260131 03:09:02.170732 31057 webserver.cc:492] Webserver started at http://127.19.84.3:33693/ using document root <none> and password file <none>
I20260131 03:09:02.170923 31057 fs_manager.cc:362] Metadata directory not provided
I20260131 03:09:02.170979 31057 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:09:02.171111 31057 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:09:02.172117 31057 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "a0dd820690994d53a98fdfd3c42ab601"
format_stamp: "Formatted at 2026-01-31 03:09:02 on dist-test-slave-5nnp"
server_key: "154768eef4ae5215172de4f0f185ce1b"
server_key_iv: "4bd14a8989b9f78579f098518251b318"
server_key_version: "encryptionkey@0"
I20260131 03:09:02.172446 31057 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "a0dd820690994d53a98fdfd3c42ab601"
format_stamp: "Formatted at 2026-01-31 03:09:02 on dist-test-slave-5nnp"
server_key: "154768eef4ae5215172de4f0f185ce1b"
server_key_iv: "4bd14a8989b9f78579f098518251b318"
server_key_version: "encryptionkey@0"
I20260131 03:09:02.173615 31057 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260131 03:09:02.174315 31072 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:09:02.174484 31057 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260131 03:09:02.174567 31057 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "a0dd820690994d53a98fdfd3c42ab601"
format_stamp: "Formatted at 2026-01-31 03:09:02 on dist-test-slave-5nnp"
server_key: "154768eef4ae5215172de4f0f185ce1b"
server_key_iv: "4bd14a8989b9f78579f098518251b318"
server_key_version: "encryptionkey@0"
I20260131 03:09:02.174641 31057 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:09:02.194389 31057 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:09:02.196508 31057 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:09:02.196669 31057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:09:02.196920 31057 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:09:02.197259 31057 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:09:02.197314 31057 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:09:02.197355 31057 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:09:02.197398 31057 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:09:02.203673 31057 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:46787
I20260131 03:09:02.203716 31185 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:46787 every 8 connection(s)
I20260131 03:09:02.204044 31057 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
Jan 31 03:09:02 dist-test-slave-5nnp krb5kdc[30673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828942, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:09:02.212440 30760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:09:02.204806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:57045 (local address 127.19.84.62:45169)
0131 03:09:02.204851 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0131 03:09:02.204853 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:09:02.205032 (+ 179us) server_negotiation.cc:408] Connection header received
0131 03:09:02.205502 (+ 470us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:09:02.205503 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:09:02.205515 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:09:02.205561 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:09:02.206257 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.206744 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:09:02.207261 (+ 517us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.207389 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:09:02.209685 (+ 2296us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:09:02.209689 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:09:02.209690 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:09:02.209697 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:09:02.211046 (+ 1349us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:09:02.211539 (+ 493us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:09:02.211541 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:09:02.211541 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:09:02.211574 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:09:02.211798 (+ 224us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:09:02.211799 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:09:02.211799 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:09:02.211901 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:09:02.211977 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0131 03:09:02.212239 (+ 262us) server_negotiation.cc:300] Negotiation successful
0131 03:09:02.212343 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260131 03:09:02.212564 31188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:09:02.204749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:45169 (local address 127.19.84.3:57045)
0131 03:09:02.204919 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0131 03:09:02.204947 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0131 03:09:02.205411 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:09:02.205613 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:09:02.205616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:09:02.205905 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:09:02.206184 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:09:02.206188 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.206817 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:09:02.206818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:09:02.207189 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:09:02.207190 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.207291 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:09:02.208184 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:09:02.208201 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:09:02.209589 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:09:02.211168 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:09:02.211175 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:09:02.211178 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:09:02.211446 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:09:02.211645 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:09:02.211646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:09:02.211647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:09:02.211733 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:09:02.211965 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:09:02.211970 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:09:02.212107 (+ 137us) client_negotiation.cc:769] Sending connection context
0131 03:09:02.212228 (+ 121us) client_negotiation.cc:240] Negotiation successful
0131 03:09:02.212361 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
I20260131 03:09:02.212787 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 31057
I20260131 03:09:02.212872 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:09:02.213078 19792 external_mini_cluster.cc:1468] Setting key 3f6d42c4de84783f3d07cedadbafe431
I20260131 03:09:02.213066 31186 heartbeater.cc:344] Connected to a master server at 127.19.84.62:45169
I20260131 03:09:02.213158 31186 heartbeater.cc:461] Registering TS with master...
I20260131 03:09:02.213376 31186 heartbeater.cc:507] Master 127.19.84.62:45169 requested a full tablet report, sending...
I20260131 03:09:02.213768 30720 ts_manager.cc:194] Registered new tserver with Master: a0dd820690994d53a98fdfd3c42ab601 (127.19.84.3:46787)
I20260131 03:09:02.214355 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:09:02.214468 30720 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:57045
I20260131 03:09:02.221359 30760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:09:02.215866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45180 (local address 127.19.84.62:45169)
0131 03:09:02.215929 (+ 63us) server_negotiation.cc:207] Beginning negotiation
0131 03:09:02.215932 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:09:02.216041 (+ 109us) server_negotiation.cc:408] Connection header received
0131 03:09:02.216138 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:09:02.216140 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:09:02.216167 (+ 27us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:09:02.216224 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:09:02.216901 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.217364 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:09:02.217936 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.218129 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:09:02.219073 (+ 944us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:09:02.219076 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:09:02.219077 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:09:02.219084 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:09:02.220297 (+ 1213us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:09:02.220633 (+ 336us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:09:02.220634 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:09:02.220635 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:09:02.220662 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:09:02.220823 (+ 161us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:09:02.220824 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:09:02.220824 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:09:02.220929 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:09:02.221033 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0131 03:09:02.221146 (+ 113us) server_negotiation.cc:300] Negotiation successful
0131 03:09:02.221223 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260131 03:09:02.222815 30720 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:09:02.224099 30720 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:09:02.229439 31197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:09:02.226515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:38137 (local address 127.0.0.1:41694)
0131 03:09:02.226797 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0131 03:09:02.226808 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:09:02.226871 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:09:02.227081 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:09:02.227085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:09:02.227088 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:09:02.227349 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:09:02.227352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.228391 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:09:02.228392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:09:02.229152 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:09:02.229153 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.229228 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:09:02.229274 (+ 46us) client_negotiation.cc:769] Sending connection context
0131 03:09:02.229324 (+ 50us) client_negotiation.cc:240] Negotiation successful
0131 03:09:02.229363 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":97,"threads_started":1}
I20260131 03:09:02.229912 31199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:09:02.226432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:36389 (local address 127.0.0.1:59906)
0131 03:09:02.226679 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0131 03:09:02.226710 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0131 03:09:02.226814 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:09:02.227154 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:09:02.227155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:09:02.227157 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:09:02.227352 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:09:02.227353 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.228809 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:09:02.228810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:09:02.229646 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:09:02.229649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.229708 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:09:02.229712 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:09:02.229742 (+ 30us) client_negotiation.cc:240] Negotiation successful
0131 03:09:02.229783 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":71,"threads_started":1}
I20260131 03:09:02.230186 31200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:09:02.226622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41694 (local address 127.19.84.1:38137)
0131 03:09:02.226829 (+ 207us) server_negotiation.cc:207] Beginning negotiation
0131 03:09:02.226842 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0131 03:09:02.226859 (+ 17us) server_negotiation.cc:408] Connection header received
0131 03:09:02.226956 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:09:02.226959 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:09:02.226977 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:09:02.227043 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:09:02.227468 (+ 425us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.228298 (+ 830us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:09:02.229263 (+ 965us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.229978 (+ 715us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:09:02.230031 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0131 03:09:02.230060 (+ 29us) server_negotiation.cc:300] Negotiation successful
0131 03:09:02.230107 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":112,"threads_started":1}
I20260131 03:09:02.230248 31198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:09:02.226515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:46787 (local address 127.0.0.1:46792)
0131 03:09:02.226900 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0131 03:09:02.226910 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:09:02.226950 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:09:02.227527 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:09:02.227528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:09:02.227530 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:09:02.227728 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:09:02.227729 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.228955 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:09:02.228956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:09:02.230024 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:09:02.230027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.230094 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:09:02.230096 (+ 2us) client_negotiation.cc:769] Sending connection context
0131 03:09:02.230126 (+ 30us) client_negotiation.cc:240] Negotiation successful
0131 03:09:02.230162 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":10,"thread_start_us":87,"threads_started":1}
I20260131 03:09:02.230870 31201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:09:02.226660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59906 (local address 127.19.84.2:36389)
0131 03:09:02.226968 (+ 308us) server_negotiation.cc:207] Beginning negotiation
0131 03:09:02.226975 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:09:02.226984 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:09:02.227045 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:09:02.227047 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:09:02.227065 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:09:02.227136 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:09:02.227482 (+ 346us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.228720 (+ 1238us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:09:02.230132 (+ 1412us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.230672 (+ 540us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:09:02.230713 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0131 03:09:02.230743 (+ 30us) server_negotiation.cc:300] Negotiation successful
0131 03:09:02.230785 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":73,"threads_started":1}
I20260131 03:09:02.230821 30848 tablet_service.cc:1505] Processing CreateTablet for tablet 71323172d4a7435895fbed1b3fa8b604 (DEFAULT_TABLE table=test-table [id=dc6ab26ed38842ddbb2a9ff98722faae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:09:02.230942 31202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:09:02.226804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46792 (local address 127.19.84.3:46787)
0131 03:09:02.227274 (+ 470us) server_negotiation.cc:207] Beginning negotiation
0131 03:09:02.227280 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:09:02.227289 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:09:02.227343 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:09:02.227344 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:09:02.227360 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:09:02.227443 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:09:02.227806 (+ 363us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.228872 (+ 1066us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:09:02.230411 (+ 1539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.230785 (+ 374us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:09:02.230816 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
0131 03:09:02.230838 (+ 22us) server_negotiation.cc:300] Negotiation successful
0131 03:09:02.230864 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":70,"threads_started":1}
I20260131 03:09:02.231166 30848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71323172d4a7435895fbed1b3fa8b604. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:09:02.231484 30984 tablet_service.cc:1505] Processing CreateTablet for tablet 71323172d4a7435895fbed1b3fa8b604 (DEFAULT_TABLE table=test-table [id=dc6ab26ed38842ddbb2a9ff98722faae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:09:02.231745 30984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71323172d4a7435895fbed1b3fa8b604. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:09:02.231932 31120 tablet_service.cc:1505] Processing CreateTablet for tablet 71323172d4a7435895fbed1b3fa8b604 (DEFAULT_TABLE table=test-table [id=dc6ab26ed38842ddbb2a9ff98722faae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:09:02.232172 31120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71323172d4a7435895fbed1b3fa8b604. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:09:02.233525 31203 tablet_bootstrap.cc:492] T 71323172d4a7435895fbed1b3fa8b604 P 838c2ab5e1d14bcd9435831f92d269c9: Bootstrap starting.
I20260131 03:09:02.234028 31203 tablet_bootstrap.cc:654] T 71323172d4a7435895fbed1b3fa8b604 P 838c2ab5e1d14bcd9435831f92d269c9: Neither blocks nor log segments found. Creating new log.
I20260131 03:09:02.234239 31203 log.cc:826] T 71323172d4a7435895fbed1b3fa8b604 P 838c2ab5e1d14bcd9435831f92d269c9: Log is configured to *not* fsync() on all Append() calls
I20260131 03:09:02.234498 31204 tablet_bootstrap.cc:492] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539: Bootstrap starting.
I20260131 03:09:02.235050 31206 tablet_bootstrap.cc:492] T 71323172d4a7435895fbed1b3fa8b604 P a0dd820690994d53a98fdfd3c42ab601: Bootstrap starting.
I20260131 03:09:02.235133 31204 tablet_bootstrap.cc:654] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539: Neither blocks nor log segments found. Creating new log.
I20260131 03:09:02.235483 31204 log.cc:826] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539: Log is configured to *not* fsync() on all Append() calls
I20260131 03:09:02.235996 31203 tablet_bootstrap.cc:492] T 71323172d4a7435895fbed1b3fa8b604 P 838c2ab5e1d14bcd9435831f92d269c9: No bootstrap required, opened a new log
I20260131 03:09:02.236066 31203 ts_tablet_manager.cc:1403] T 71323172d4a7435895fbed1b3fa8b604 P 838c2ab5e1d14bcd9435831f92d269c9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260131 03:09:02.236140 31204 tablet_bootstrap.cc:492] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539: No bootstrap required, opened a new log
I20260131 03:09:02.236217 31204 ts_tablet_manager.cc:1403] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:09:02.237658 31203 raft_consensus.cc:359] T 71323172d4a7435895fbed1b3fa8b604 P 838c2ab5e1d14bcd9435831f92d269c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cde00d385549c499d650b62353e539" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 36389 } } peers { permanent_uuid: "a0dd820690994d53a98fdfd3c42ab601" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 46787 } } peers { permanent_uuid: "838c2ab5e1d14bcd9435831f92d269c9" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38137 } }
I20260131 03:09:02.237658 31204 raft_consensus.cc:359] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cde00d385549c499d650b62353e539" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 36389 } } peers { permanent_uuid: "a0dd820690994d53a98fdfd3c42ab601" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 46787 } } peers { permanent_uuid: "838c2ab5e1d14bcd9435831f92d269c9" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38137 } }
I20260131 03:09:02.237809 31204 raft_consensus.cc:385] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:09:02.237809 31203 raft_consensus.cc:385] T 71323172d4a7435895fbed1b3fa8b604 P 838c2ab5e1d14bcd9435831f92d269c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:09:02.237841 31203 raft_consensus.cc:740] T 71323172d4a7435895fbed1b3fa8b604 P 838c2ab5e1d14bcd9435831f92d269c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 838c2ab5e1d14bcd9435831f92d269c9, State: Initialized, Role: FOLLOWER
I20260131 03:09:02.237841 31204 raft_consensus.cc:740] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68cde00d385549c499d650b62353e539, State: Initialized, Role: FOLLOWER
I20260131 03:09:02.237931 31204 consensus_queue.cc:260] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cde00d385549c499d650b62353e539" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 36389 } } peers { permanent_uuid: "a0dd820690994d53a98fdfd3c42ab601" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 46787 } } peers { permanent_uuid: "838c2ab5e1d14bcd9435831f92d269c9" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38137 } }
I20260131 03:09:02.237931 31203 consensus_queue.cc:260] T 71323172d4a7435895fbed1b3fa8b604 P 838c2ab5e1d14bcd9435831f92d269c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cde00d385549c499d650b62353e539" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 36389 } } peers { permanent_uuid: "a0dd820690994d53a98fdfd3c42ab601" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 46787 } } peers { permanent_uuid: "838c2ab5e1d14bcd9435831f92d269c9" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38137 } }
I20260131 03:09:02.238169 31203 ts_tablet_manager.cc:1434] T 71323172d4a7435895fbed1b3fa8b604 P 838c2ab5e1d14bcd9435831f92d269c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:09:02.238219 30914 heartbeater.cc:499] Master 127.19.84.62:45169 was elected leader, sending a full tablet report...
I20260131 03:09:02.238299 31206 tablet_bootstrap.cc:654] T 71323172d4a7435895fbed1b3fa8b604 P a0dd820690994d53a98fdfd3c42ab601: Neither blocks nor log segments found. Creating new log.
I20260131 03:09:02.238169 31204 ts_tablet_manager.cc:1434] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:09:02.238416 31050 heartbeater.cc:499] Master 127.19.84.62:45169 was elected leader, sending a full tablet report...
I20260131 03:09:02.238637 31206 log.cc:826] T 71323172d4a7435895fbed1b3fa8b604 P a0dd820690994d53a98fdfd3c42ab601: Log is configured to *not* fsync() on all Append() calls
I20260131 03:09:02.239447 31206 tablet_bootstrap.cc:492] T 71323172d4a7435895fbed1b3fa8b604 P a0dd820690994d53a98fdfd3c42ab601: No bootstrap required, opened a new log
I20260131 03:09:02.239531 31206 ts_tablet_manager.cc:1403] T 71323172d4a7435895fbed1b3fa8b604 P a0dd820690994d53a98fdfd3c42ab601: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260131 03:09:02.240867 31206 raft_consensus.cc:359] T 71323172d4a7435895fbed1b3fa8b604 P a0dd820690994d53a98fdfd3c42ab601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cde00d385549c499d650b62353e539" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 36389 } } peers { permanent_uuid: "a0dd820690994d53a98fdfd3c42ab601" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 46787 } } peers { permanent_uuid: "838c2ab5e1d14bcd9435831f92d269c9" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38137 } }
I20260131 03:09:02.241005 31206 raft_consensus.cc:385] T 71323172d4a7435895fbed1b3fa8b604 P a0dd820690994d53a98fdfd3c42ab601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:09:02.241068 31206 raft_consensus.cc:740] T 71323172d4a7435895fbed1b3fa8b604 P a0dd820690994d53a98fdfd3c42ab601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0dd820690994d53a98fdfd3c42ab601, State: Initialized, Role: FOLLOWER
I20260131 03:09:02.241178 31206 consensus_queue.cc:260] T 71323172d4a7435895fbed1b3fa8b604 P a0dd820690994d53a98fdfd3c42ab601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cde00d385549c499d650b62353e539" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 36389 } } peers { permanent_uuid: "a0dd820690994d53a98fdfd3c42ab601" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 46787 } } peers { permanent_uuid: "838c2ab5e1d14bcd9435831f92d269c9" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38137 } }
I20260131 03:09:02.241405 31206 ts_tablet_manager.cc:1434] T 71323172d4a7435895fbed1b3fa8b604 P a0dd820690994d53a98fdfd3c42ab601: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260131 03:09:02.241480 31186 heartbeater.cc:499] Master 127.19.84.62:45169 was elected leader, sending a full tablet report...
W20260131 03:09:02.270087 31051 tablet.cc:2378] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260131 03:09:02.347302 30915 tablet.cc:2378] T 71323172d4a7435895fbed1b3fa8b604 P 838c2ab5e1d14bcd9435831f92d269c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260131 03:09:02.454775 31187 tablet.cc:2378] T 71323172d4a7435895fbed1b3fa8b604 P a0dd820690994d53a98fdfd3c42ab601: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:09:02.520630 31208 raft_consensus.cc:493] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:09:02.520756 31208 raft_consensus.cc:515] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cde00d385549c499d650b62353e539" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 36389 } } peers { permanent_uuid: "a0dd820690994d53a98fdfd3c42ab601" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 46787 } } peers { permanent_uuid: "838c2ab5e1d14bcd9435831f92d269c9" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38137 } }
I20260131 03:09:02.521070 31208 leader_election.cc:290] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0dd820690994d53a98fdfd3c42ab601 (127.19.84.3:46787), 838c2ab5e1d14bcd9435831f92d269c9 (127.19.84.1:38137)
I20260131 03:09:02.523206 31052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:09:02.521273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:38137 (local address 127.19.84.2:55993)
0131 03:09:02.521316 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0131 03:09:02.521329 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:09:02.521382 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:09:02.521559 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:09:02.521560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:09:02.521562 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:09:02.521754 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:09:02.521756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.522403 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:09:02.522405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:09:02.522968 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:09:02.522970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.523062 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:09:02.523066 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:09:02.523098 (+ 32us) client_negotiation.cc:240] Negotiation successful
0131 03:09:02.523134 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260131 03:09:02.523448 31212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:09:02.521273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:46787 (local address 127.19.84.2:56229)
0131 03:09:02.521433 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0131 03:09:02.521442 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:09:02.521508 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:09:02.521647 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:09:02.521648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:09:02.521650 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:09:02.521910 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:09:02.521911 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.522586 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:09:02.522587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:09:02.523186 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:09:02.523188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.523280 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:09:02.523284 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:09:02.523317 (+ 33us) client_negotiation.cc:240] Negotiation successful
0131 03:09:02.523352 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":57,"thread_start_us":63,"threads_started":1}
I20260131 03:09:02.523741 31200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:09:02.521321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:55993 (local address 127.19.84.1:38137)
0131 03:09:02.521368 (+ 47us) server_negotiation.cc:207] Beginning negotiation
0131 03:09:02.521371 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:09:02.521382 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:09:02.521451 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:09:02.521452 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:09:02.521461 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:09:02.521552 (+ 91us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:09:02.521839 (+ 287us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.522316 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:09:02.523060 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.523601 (+ 541us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:09:02.523610 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0131 03:09:02.523638 (+ 28us) server_negotiation.cc:300] Negotiation successful
0131 03:09:02.523675 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260131 03:09:02.523883 31202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:09:02.521335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:56229 (local address 127.19.84.3:46787)
0131 03:09:02.521379 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0131 03:09:02.521381 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:09:02.521522 (+ 141us) server_negotiation.cc:408] Connection header received
0131 03:09:02.521563 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:09:02.521565 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:09:02.521573 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:09:02.521644 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:09:02.522018 (+ 374us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.522499 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:09:02.523280 (+ 781us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.523743 (+ 463us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:09:02.523749 (+ 6us) server_negotiation.cc:1036] Waiting for connection context
0131 03:09:02.523768 (+ 19us) server_negotiation.cc:300] Negotiation successful
0131 03:09:02.523790 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260131 03:09:02.524082 30868 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71323172d4a7435895fbed1b3fa8b604" candidate_uuid: "68cde00d385549c499d650b62353e539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "838c2ab5e1d14bcd9435831f92d269c9" is_pre_election: true
I20260131 03:09:02.524196 31140 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71323172d4a7435895fbed1b3fa8b604" candidate_uuid: "68cde00d385549c499d650b62353e539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0dd820690994d53a98fdfd3c42ab601" is_pre_election: true
I20260131 03:09:02.524256 30868 raft_consensus.cc:2468] T 71323172d4a7435895fbed1b3fa8b604 P 838c2ab5e1d14bcd9435831f92d269c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68cde00d385549c499d650b62353e539 in term 0.
I20260131 03:09:02.524353 31140 raft_consensus.cc:2468] T 71323172d4a7435895fbed1b3fa8b604 P a0dd820690994d53a98fdfd3c42ab601 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68cde00d385549c499d650b62353e539 in term 0.
I20260131 03:09:02.524508 30939 leader_election.cc:304] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68cde00d385549c499d650b62353e539, 838c2ab5e1d14bcd9435831f92d269c9; no voters:
I20260131 03:09:02.524657 31208 raft_consensus.cc:2804] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:09:02.524739 31208 raft_consensus.cc:493] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:09:02.524776 31208 raft_consensus.cc:3060] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:09:02.525453 31208 raft_consensus.cc:515] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cde00d385549c499d650b62353e539" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 36389 } } peers { permanent_uuid: "a0dd820690994d53a98fdfd3c42ab601" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 46787 } } peers { permanent_uuid: "838c2ab5e1d14bcd9435831f92d269c9" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38137 } }
I20260131 03:09:02.525631 31208 leader_election.cc:290] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539 [CANDIDATE]: Term 1 election: Requested vote from peers a0dd820690994d53a98fdfd3c42ab601 (127.19.84.3:46787), 838c2ab5e1d14bcd9435831f92d269c9 (127.19.84.1:38137)
I20260131 03:09:02.525758 30868 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71323172d4a7435895fbed1b3fa8b604" candidate_uuid: "68cde00d385549c499d650b62353e539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "838c2ab5e1d14bcd9435831f92d269c9"
I20260131 03:09:02.525753 31140 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71323172d4a7435895fbed1b3fa8b604" candidate_uuid: "68cde00d385549c499d650b62353e539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0dd820690994d53a98fdfd3c42ab601"
I20260131 03:09:02.525830 31140 raft_consensus.cc:3060] T 71323172d4a7435895fbed1b3fa8b604 P a0dd820690994d53a98fdfd3c42ab601 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:09:02.525830 30868 raft_consensus.cc:3060] T 71323172d4a7435895fbed1b3fa8b604 P 838c2ab5e1d14bcd9435831f92d269c9 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:09:02.526705 30868 raft_consensus.cc:2468] T 71323172d4a7435895fbed1b3fa8b604 P 838c2ab5e1d14bcd9435831f92d269c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68cde00d385549c499d650b62353e539 in term 1.
I20260131 03:09:02.526705 31140 raft_consensus.cc:2468] T 71323172d4a7435895fbed1b3fa8b604 P a0dd820690994d53a98fdfd3c42ab601 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68cde00d385549c499d650b62353e539 in term 1.
I20260131 03:09:02.526897 30937 leader_election.cc:304] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68cde00d385549c499d650b62353e539, a0dd820690994d53a98fdfd3c42ab601; no voters:
I20260131 03:09:02.527014 31208 raft_consensus.cc:2804] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:09:02.527196 31208 raft_consensus.cc:697] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539 [term 1 LEADER]: Becoming Leader. State: Replica: 68cde00d385549c499d650b62353e539, State: Running, Role: LEADER
I20260131 03:09:02.527303 31208 consensus_queue.cc:237] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cde00d385549c499d650b62353e539" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 36389 } } peers { permanent_uuid: "a0dd820690994d53a98fdfd3c42ab601" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 46787 } } peers { permanent_uuid: "838c2ab5e1d14bcd9435831f92d269c9" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38137 } }
I20260131 03:09:02.528100 30720 catalog_manager.cc:5654] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68cde00d385549c499d650b62353e539 (127.19.84.2). New cstate: current_term: 1 leader_uuid: "68cde00d385549c499d650b62353e539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68cde00d385549c499d650b62353e539" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0dd820690994d53a98fdfd3c42ab601" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 46787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "838c2ab5e1d14bcd9435831f92d269c9" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38137 } health_report { overall_health: UNKNOWN } } }
I20260131 03:09:02.582628 31201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:09:02.580507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59918 (local address 127.19.84.2:36389)
0131 03:09:02.580587 (+ 80us) server_negotiation.cc:207] Beginning negotiation
0131 03:09:02.580589 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:09:02.580599 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:09:02.580645 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:09:02.580646 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:09:02.580662 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:09:02.580730 (+ 68us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:09:02.580990 (+ 260us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.581486 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:09:02.582074 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:09:02.582236 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:09:02.582270 (+ 34us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:09:02.582358 (+ 88us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:09:02.582446 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0131 03:09:02.582520 (+ 74us) server_negotiation.cc:300] Negotiation successful
0131 03:09:02.582544 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260131 03:09:02.584352 30868 raft_consensus.cc:1275] T 71323172d4a7435895fbed1b3fa8b604 P 838c2ab5e1d14bcd9435831f92d269c9 [term 1 FOLLOWER]: Refusing update from remote peer 68cde00d385549c499d650b62353e539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:09:02.584352 31140 raft_consensus.cc:1275] T 71323172d4a7435895fbed1b3fa8b604 P a0dd820690994d53a98fdfd3c42ab601 [term 1 FOLLOWER]: Refusing update from remote peer 68cde00d385549c499d650b62353e539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:09:02.584712 31213 consensus_queue.cc:1048] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0dd820690994d53a98fdfd3c42ab601" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 46787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:09:02.584825 31208 consensus_queue.cc:1048] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838c2ab5e1d14bcd9435831f92d269c9" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 38137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:09:02.587047 31216 mvcc.cc:204] Tried to move back new op lower bound from 7249219348822974464 to 7249219348592484352. Current Snapshot: MvccSnapshot[applied={T|T < 7249219348822974464}]
I20260131 03:09:02.589680 30719 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:09:02.589803 30719 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:09:02.590586 30719 catalog_manager.cc:5941] T 00000000000000000000000000000000 P ef8a283c14254471919ad837154b9609: Sending DeleteTablet for 3 replicas of tablet 71323172d4a7435895fbed1b3fa8b604
I20260131 03:09:02.590929 30848 tablet_service.cc:1552] Processing DeleteTablet for tablet 71323172d4a7435895fbed1b3fa8b604 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:09:02 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:41694
I20260131 03:09:02.590929 31120 tablet_service.cc:1552] Processing DeleteTablet for tablet 71323172d4a7435895fbed1b3fa8b604 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:09:02 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:46792
I20260131 03:09:02.591148 30984 tablet_service.cc:1552] Processing DeleteTablet for tablet 71323172d4a7435895fbed1b3fa8b604 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:09:02 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:59906
I20260131 03:09:02.591189 31227 tablet_replica.cc:333] T 71323172d4a7435895fbed1b3fa8b604 P a0dd820690994d53a98fdfd3c42ab601: stopping tablet replica
I20260131 03:09:02.591284 31227 raft_consensus.cc:2243] T 71323172d4a7435895fbed1b3fa8b604 P a0dd820690994d53a98fdfd3c42ab601 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:09:02.591408 31227 raft_consensus.cc:2272] T 71323172d4a7435895fbed1b3fa8b604 P a0dd820690994d53a98fdfd3c42ab601 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:09:02.591500 31228 tablet_replica.cc:333] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539: stopping tablet replica
I20260131 03:09:02.591567 31226 tablet_replica.cc:333] T 71323172d4a7435895fbed1b3fa8b604 P 838c2ab5e1d14bcd9435831f92d269c9: stopping tablet replica
I20260131 03:09:02.591650 31226 raft_consensus.cc:2243] T 71323172d4a7435895fbed1b3fa8b604 P 838c2ab5e1d14bcd9435831f92d269c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:09:02.591742 31226 raft_consensus.cc:2272] T 71323172d4a7435895fbed1b3fa8b604 P 838c2ab5e1d14bcd9435831f92d269c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:09:02.591789 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 30785
I20260131 03:09:02.593014 31228 raft_consensus.cc:2243] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:09:02.593156 31228 raft_consensus.cc:2272] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:09:02.593552 31228 ts_tablet_manager.cc:1916] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:09:02.594947 31228 ts_tablet_manager.cc:1929] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:09:02.595029 31228 log.cc:1199] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-1/wal/wals/71323172d4a7435895fbed1b3fa8b604
I20260131 03:09:02.595265 31228 ts_tablet_manager.cc:1950] T 71323172d4a7435895fbed1b3fa8b604 P 68cde00d385549c499d650b62353e539: Deleting consensus metadata
I20260131 03:09:02.595652 30708 catalog_manager.cc:4985] TS 68cde00d385549c499d650b62353e539 (127.19.84.2:36389): tablet 71323172d4a7435895fbed1b3fa8b604 (table test-table [id=dc6ab26ed38842ddbb2a9ff98722faae]) successfully deleted
I20260131 03:09:02.596240 31227 ts_tablet_manager.cc:1916] T 71323172d4a7435895fbed1b3fa8b604 P a0dd820690994d53a98fdfd3c42ab601: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260131 03:09:02.597349 30707 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.84.1:38137 (error 108)
I20260131 03:09:02.597522 31227 ts_tablet_manager.cc:1929] T 71323172d4a7435895fbed1b3fa8b604 P a0dd820690994d53a98fdfd3c42ab601: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:09:02.597651 31227 log.cc:1199] T 71323172d4a7435895fbed1b3fa8b604 P a0dd820690994d53a98fdfd3c42ab601: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/71323172d4a7435895fbed1b3fa8b604
I20260131 03:09:02.598125 31227 ts_tablet_manager.cc:1950] T 71323172d4a7435895fbed1b3fa8b604 P a0dd820690994d53a98fdfd3c42ab601: Deleting consensus metadata
I20260131 03:09:02.598141 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 30921
I20260131 03:09:02.602491 30705 catalog_manager.cc:4985] TS a0dd820690994d53a98fdfd3c42ab601 (127.19.84.3:46787): tablet 71323172d4a7435895fbed1b3fa8b604 (table test-table [id=dc6ab26ed38842ddbb2a9ff98722faae]) successfully deleted
I20260131 03:09:02.602849 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 31057
I20260131 03:09:02.603492 31198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:09:02.602912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:38137 (local address 127.0.0.1:41698)
0131 03:09:02.603411 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0131 03:09:02.603450 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.1:38137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476}
W20260131 03:09:02.603717 30707 catalog_manager.cc:4712] TS 838c2ab5e1d14bcd9435831f92d269c9 (127.19.84.1:38137): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 71323172d4a7435895fbed1b3fa8b604: Network error: Client connection negotiation failed: client connection to 127.19.84.1:38137: connect: Connection refused (error 111)
I20260131 03:09:02.608222 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 30689
2026-01-31T03:09:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3262 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:09:02 dist-test-slave-5nnp krb5kdc[31232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:09:02 dist-test-slave-5nnp krb5kdc[31232](info): set up 2 sockets
Jan 31 03:09:02 dist-test-slave-5nnp krb5kdc[31232](info): commencing operation
krb5kdc: starting...
W20260131 03:09:04.608362 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.980s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:09:04 dist-test-slave-5nnp krb5kdc[31232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769828944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:09:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:09:04Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.84.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.84.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.84.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828849314684-19792-0/krb5kdc/rangeradmin_127.19.84.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.84.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828849314684-19792-0/krb5kdc/rangeradmin_127.19.84.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.84.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.84.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.84.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828849314684-19792-0/krb5kdc/rangerlookup_127.19.84.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.84.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828849314684-19792-0/krb5kdc/rangerlookup_127.19.84.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.84.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828849314684-19792-0/krb5kdc/HTTP_127.19.84.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.84.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828849314684-19792-0/krb5kdc/HTTP_127.19.84.20@KRBTEST.COM.keytab.
I20260131 03:09:04.761438 19792 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828849314684-19792-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWVHG5r/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828849314684-19792-0/minicluster-data/postgres -l logfile start
2026-01-31 03:09:07.487 UTC [31263] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-01-31 03:09:07.487 UTC [31263] LOG: listening on IPv4 address "127.19.84.20", port 45595
2026-01-31 03:09:07.491 UTC [31263] LOG: listening on Unix socket "/tmp/.s.PGSQL.45595"
2026-01-31 03:09:07.496 UTC [31268] LOG: database system was shut down at 2026-01-31 03:09:05 UTC
2026-01-31 03:09:07.501 UTC [31263] LOG: database system is ready to accept connections
I20260131 03:09:09.436126 19792 mini_postgres.cc:96] Postgres bound to 45595
2026-01-31 03:09:09.442 UTC [31274] FATAL: database "slave" does not exist
127.19.84.20:45595 - accepting connections
I20260131 03:09:09.442850 19792 mini_ranger.cc:161] Starting Ranger...
I20260131 03:09:09.460917 19792 mini_ranger.cc:84] Created miniranger Postgres user
I20260131 03:09:09.527359 19792 mini_ranger.cc:87] Created ranger Postgres database
I20260131 03:09:09.527478 19792 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828849314684-19792-0/minicluster-data/ranger-admin
2026-01-31 03:09:09,806 [I] DB FLAVOR :POSTGRES
2026-01-31 03:09:09,807 [I] --------- Verifying Ranger DB connection ---------
2026-01-31 03:09:09,807 [I] Checking connection..
2026-01-31 03:09:09,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:45595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-01-31 03:09:10,125 [I] Checking connection passed.
2026-01-31 03:09:10,126 [I] --------- Verifying version history table ---------
2026-01-31 03:09:10,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:45595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-01-31 03:09:10,449 [I] Table x_db_version_h does not exist in database ranger
2026-01-31 03:09:10,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:45595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-01-31 03:09:10,782 [I] Table x_db_version_h does not exist in database ranger
2026-01-31 03:09:10,782 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-01-31 03:09:10,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:45595/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-01-31 03:09:11.123 UTC [31341] WARNING: there is no transaction in progress
2026-01-31 03:09:11,130 [I] create_dbversion_catalog.sql file imported successfully
2026-01-31 03:09:11,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:45595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-01-31 03:09:11,471 [I] Table x_db_version_h already exists in database 'ranger'
2026-01-31 03:09:11,472 [I] --------- Importing Ranger Core DB Schema ---------
2026-01-31 03:09:11,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:45595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-01-31 03:09:11,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:45595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-01-31 03:09:12,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:45595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5nnp.c.gcp-upstream.internal','N') ;"
2026-01-31 03:09:12,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:45595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-01-31 03:09:12,690 [I] Table x_portal_user does not exist in database ranger
2026-01-31 03:09:12,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:45595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-01-31 03:09:13,024 [I] Table x_policy_ref_group does not exist in database ranger
2026-01-31 03:09:13,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:45595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-01-31 03:09:13,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:45595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-01-31 03:09:13,623 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-01-31 03:09:13,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:45595/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-01-31 03:09:14.510 UTC [31458] WARNING: there is no transaction in progress
2026-01-31 03:09:14.522 UTC [31458] WARNING: there is no transaction in progress
2026-01-31 03:09:14.534 UTC [31458] WARNING: there is no transaction in progress
2026-01-31 03:09:14.545 UTC [31458] WARNING: there is no transaction in progress
2026-01-31 03:09:14.556 UTC [31458] WARNING: there is no transaction in progress
2026-01-31 03:09:14.569 UTC [31458] WARNING: there is no transaction in progress
2026-01-31 03:09:14.620 UTC [31458] WARNING: there is no transaction in progress
2026-01-31 03:09:14.627 UTC [31458] WARNING: there is no transaction in progress
2026-01-31 03:09:14.637 UTC [31458] WARNING: there is no transaction in progress
2026-01-31 03:09:14.647 UTC [31458] WARNING: there is no transaction in progress
2026-01-31 03:09:14.656 UTC [31458] WARNING: there is no transaction in progress
2026-01-31 03:09:14.666 UTC [31458] WARNING: there is no transaction in progress
2026-01-31 03:09:14.675 UTC [31458] WARNING: there is no transaction in progress
2026-01-31 03:09:14.680 UTC [31458] WARNING: there is no transaction in progress
2026-01-31 03:09:14.684 UTC [31458] WARNING: there is no transaction in progress
2026-01-31 03:09:14.689 UTC [31458] WARNING: there is no transaction in progress
2026-01-31 03:09:15,146 [I] ranger_core_db_postgres.sql file imported successfully
2026-01-31 03:09:15,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:45595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-01-31 03:09:15,454 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-01-31 03:09:15,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:45595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-01-31 03:09:15,757 [I] Table x_portal_user already exists in database 'ranger'
2026-01-31 03:09:15,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:45595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-01-31 03:09:16,091 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-01-31 03:09:16,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:45595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-01-31 03:09:16,404 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:45595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-01-31 03:09:16,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:45595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5nnp.c.gcp-upstream.internal';"
2026-01-31 03:09:17,032 [I] CORE_DB_SCHEMA import status has been updated
2026-01-31 03:09:17,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:45595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-01-31 03:09:17,371 [I] DB_PATCHES have already been applied
I20260131 03:09:17.377102 19792 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828849314684-19792-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWVHG5r/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260131 03:09:17.377175 19792 mini_ranger.cc:193] Using host: 127.19.84.20
I20260131 03:09:17.379885 19792 mini_ranger.cc:227] Ranger admin URL: http://127.19.84.20:38775
Jan 31, 2026 3:09:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jan 31, 2026 3:09:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jan 31, 2026 3:09:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828849314684-19792-0/minicluster-data/ranger-admin/ews/webapp
Jan 31, 2026 3:09:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828849314684-19792-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jan 31, 2026 3:09:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828849314684-19792-0/minicluster-data/ranger-admin/ews/webapp] .....
Jan 31, 2026 3:09:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828849314684-19792-0/minicluster-data/ranger-admin/ews/webapp].
Jan 31, 2026 3:09:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Jan 31, 2026 3:09:43 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jan 31, 2026 3:09:43 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.MetricsREST
Jan 31, 2026 3:09:43 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jan 31, 2026 3:09:43 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jan 31, 2026 3:09:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jan 31, 2026 3:09:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jan 31, 2026 3:09:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jan 31, 2026 3:09:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jan 31, 2026 3:09:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jan 31, 2026 3:09:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jan 31, 2026 3:09:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jan 31, 2026 3:09:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jan 31, 2026 3:09:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jan 31, 2026 3:09:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jan 31, 2026 3:09:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jan 31, 2026 3:09:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jan 31, 2026 3:09:43 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jan 31, 2026 3:09:44 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260131 03:11:18.367367 19792 mini_ranger.cc:160] Time spent starting Ranger: real 128.924s user 0.017s sys 0.074s
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/security-itest.cc:1052: Failure
Failed
Bad status: Timed out: Failed to start the Ranger service: process with PID 31552 is not yet bound to any port at the specified addresses; last attempt running lsof returned 'Not found: could not find pattern of a bound port in lsof output: p31552
f310
n127.0.0.1:37580->127.19.84.20:45595
f311
n127.0.0.1:37600->127.19.84.20:45595
f312
n127.0.0.1:37582->127.19.84.20:45595
f313
n127.0.0.1:37602->127.19.84.20:45595
f314
n127.0.0.1:37604->127.19.84.20:45595'
I20260131 03:11:18.368598 19792 mini_ranger.cc:66] Stopping Ranger...
I20260131 03:11:18.390271 19792 mini_ranger.cc:68] Stopped Ranger
2026-01-31 03:11:18.390 UTC [31263] LOG: received smart shutdown request
2026-01-31 03:11:18.394 UTC [31263] LOG: background worker "logical replication launcher" (PID 31271) exited with exit code 1
2026-01-31 03:11:18.394 UTC [31266] LOG: shutting down
2026-01-31 03:11:18.397 UTC [31266] LOG: checkpoint starting: shutdown immediate
2026-01-31 03:11:18.898 UTC [31266] LOG: checkpoint complete: wrote 1468 buffers (9.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.027 s, sync=0.469 s, total=0.504 s; sync files=815, longest=0.005 s, average=0.001 s; distance=8396 kB, estimate=8396 kB; lsn=0/1CADBA0, redo lsn=0/1CADBA0
2026-01-31 03:11:18.904 UTC [31263] LOG: database system is shut down
2026-01-31T03:11:18Z chronyd exiting
I20260131 03:11:18.918334 19792 test_util.cc:183] -----------------------------------------------
I20260131 03:11:18.918409 19792 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769828849314684-19792-0
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration (136290 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:11:18 dist-test-slave-5nnp krb5kdc[31696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:11:18 dist-test-slave-5nnp krb5kdc[31696](info): set up 2 sockets
Jan 31 03:11:18 dist-test-slave-5nnp krb5kdc[31696](info): commencing operation
krb5kdc: starting...
W20260131 03:11:20.907359 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.988s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:11:20 dist-test-slave-5nnp krb5kdc[31696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:11:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:11:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.84.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.84.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.84.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/rangeradmin_127.19.84.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.84.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/rangeradmin_127.19.84.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.84.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.84.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.84.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/rangerlookup_127.19.84.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.84.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/rangerlookup_127.19.84.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.84.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/HTTP_127.19.84.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.84.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/HTTP_127.19.84.20@KRBTEST.COM.keytab.
I20260131 03:11:21.074445 19792 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWVHG5r/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/postgres -l logfile start
2026-01-31 03:11:23.776 UTC [31727] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-01-31 03:11:23.776 UTC [31727] LOG: listening on IPv4 address "127.19.84.20", port 46689
2026-01-31 03:11:23.780 UTC [31727] LOG: listening on Unix socket "/tmp/.s.PGSQL.46689"
2026-01-31 03:11:23.785 UTC [31732] LOG: database system was shut down at 2026-01-31 03:11:22 UTC
2026-01-31 03:11:23.790 UTC [31727] LOG: database system is ready to accept connections
I20260131 03:11:25.707206 19792 mini_postgres.cc:96] Postgres bound to 46689
2026-01-31 03:11:25.712 UTC [31738] FATAL: database "slave" does not exist
127.19.84.20:46689 - accepting connections
I20260131 03:11:25.713457 19792 mini_ranger.cc:161] Starting Ranger...
I20260131 03:11:25.731493 19792 mini_ranger.cc:84] Created miniranger Postgres user
I20260131 03:11:25.803028 19792 mini_ranger.cc:87] Created ranger Postgres database
I20260131 03:11:25.803144 19792 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ranger-admin
2026-01-31 03:11:26,086 [I] DB FLAVOR :POSTGRES
2026-01-31 03:11:26,087 [I] --------- Verifying Ranger DB connection ---------
2026-01-31 03:11:26,087 [I] Checking connection..
2026-01-31 03:11:26,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:46689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-01-31 03:11:26,402 [I] Checking connection passed.
2026-01-31 03:11:26,402 [I] --------- Verifying version history table ---------
2026-01-31 03:11:26,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:46689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-01-31 03:11:26,707 [I] Table x_db_version_h does not exist in database ranger
2026-01-31 03:11:26,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:46689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-01-31 03:11:27,032 [I] Table x_db_version_h does not exist in database ranger
2026-01-31 03:11:27,032 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-01-31 03:11:27,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:46689/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-01-31 03:11:27.343 UTC [31805] WARNING: there is no transaction in progress
2026-01-31 03:11:27,350 [I] create_dbversion_catalog.sql file imported successfully
2026-01-31 03:11:27,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:46689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-01-31 03:11:27,655 [I] Table x_db_version_h already exists in database 'ranger'
2026-01-31 03:11:27,655 [I] --------- Importing Ranger Core DB Schema ---------
2026-01-31 03:11:27,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:46689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-01-31 03:11:27,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:46689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-01-31 03:11:28,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:46689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5nnp.c.gcp-upstream.internal','N') ;"
2026-01-31 03:11:28,586 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:46689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-01-31 03:11:28,881 [I] Table x_portal_user does not exist in database ranger
2026-01-31 03:11:28,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:46689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-01-31 03:11:29,203 [I] Table x_policy_ref_group does not exist in database ranger
2026-01-31 03:11:29,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:46689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-01-31 03:11:29,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:46689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-01-31 03:11:29,827 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-01-31 03:11:29,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:46689/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-01-31 03:11:30.686 UTC [31922] WARNING: there is no transaction in progress
2026-01-31 03:11:30.697 UTC [31922] WARNING: there is no transaction in progress
2026-01-31 03:11:30.709 UTC [31922] WARNING: there is no transaction in progress
2026-01-31 03:11:30.721 UTC [31922] WARNING: there is no transaction in progress
2026-01-31 03:11:30.732 UTC [31922] WARNING: there is no transaction in progress
2026-01-31 03:11:30.746 UTC [31922] WARNING: there is no transaction in progress
2026-01-31 03:11:30.807 UTC [31922] WARNING: there is no transaction in progress
2026-01-31 03:11:30.813 UTC [31922] WARNING: there is no transaction in progress
2026-01-31 03:11:30.826 UTC [31922] WARNING: there is no transaction in progress
2026-01-31 03:11:30.837 UTC [31922] WARNING: there is no transaction in progress
2026-01-31 03:11:30.847 UTC [31922] WARNING: there is no transaction in progress
2026-01-31 03:11:30.857 UTC [31922] WARNING: there is no transaction in progress
2026-01-31 03:11:30.865 UTC [31922] WARNING: there is no transaction in progress
2026-01-31 03:11:30.870 UTC [31922] WARNING: there is no transaction in progress
2026-01-31 03:11:30.873 UTC [31922] WARNING: there is no transaction in progress
2026-01-31 03:11:30.876 UTC [31922] WARNING: there is no transaction in progress
2026-01-31 03:11:31,347 [I] ranger_core_db_postgres.sql file imported successfully
2026-01-31 03:11:31,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:46689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-01-31 03:11:31,646 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-01-31 03:11:31,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:46689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-01-31 03:11:31,950 [I] Table x_portal_user already exists in database 'ranger'
2026-01-31 03:11:31,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:46689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-01-31 03:11:32,291 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-01-31 03:11:32,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:46689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-01-31 03:11:32,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:46689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-01-31 03:11:32,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:46689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5nnp.c.gcp-upstream.internal';"
2026-01-31 03:11:33,253 [I] CORE_DB_SCHEMA import status has been updated
2026-01-31 03:11:33,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:46689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-01-31 03:11:33,563 [I] DB_PATCHES have already been applied
I20260131 03:11:33.569169 19792 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWVHG5r/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260131 03:11:33.569365 19792 mini_ranger.cc:193] Using host: 127.19.84.20
I20260131 03:11:33.572064 19792 mini_ranger.cc:227] Ranger admin URL: http://127.19.84.20:44125
Jan 31, 2026 3:11:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jan 31, 2026 3:11:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jan 31, 2026 3:11:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ranger-admin/ews/webapp
Jan 31, 2026 3:11:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jan 31, 2026 3:11:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ranger-admin/ews/webapp] .....
Jan 31, 2026 3:11:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ranger-admin/ews/webapp].
Jan 31, 2026 3:11:34 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260131 03:11:35.827716 19792 mini_ranger.cc:160] Time spent starting Ranger: real 10.114s user 0.000s sys 0.009s
Jan 31, 2026 3:11:55 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jan 31, 2026 3:11:55 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.AssetREST
Jan 31, 2026 3:11:55 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jan 31, 2026 3:11:55 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jan 31, 2026 3:11:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jan 31, 2026 3:11:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jan 31, 2026 3:11:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jan 31, 2026 3:11:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jan 31, 2026 3:11:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jan 31, 2026 3:11:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jan 31, 2026 3:11:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jan 31, 2026 3:11:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jan 31, 2026 3:11:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jan 31, 2026 3:11:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jan 31, 2026 3:11:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jan 31, 2026 3:11:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jan 31, 2026 3:11:56 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jan 31, 2026 3:11:56 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260131 03:11:57.549536 19792 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.19.84.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.19.84.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.19.84.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/rangerkms_127.19.84.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.19.84.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/rangerkms_127.19.84.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.84.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.19.84.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/HTTP_127.19.84.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.84.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/HTTP_127.19.84.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jan 31 03:11:57 dist-test-slave-5nnp krb5kdc[31696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829117, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260131 03:11:57.636420 19792 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260131 03:11:57.646489 19792 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260131 03:11:57.770006 19792 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260131 03:11:57.770098 19792 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ranger-kms
I20260131 03:11:57.770116 19792 mini_ranger_kms.cc:227] Using postgres at 127.19.84.20:46689
2026-01-31 03:11:57,971 [I] DB FLAVOR :POSTGRES
2026-01-31 03:11:57,971 [I] --------- Verifying Ranger DB connection ---------
2026-01-31 03:11:57,971 [I] Checking connection
2026-01-31 03:11:57,971 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:46689/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-01-31 03:11:58,294 [I] connection success
2026-01-31 03:11:58,295 [I] --------- Verifying Ranger DB tables ---------
2026-01-31 03:11:58,295 [I] Verifying table ranger_masterkey in database rangerkms
2026-01-31 03:11:58,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:46689/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-01-31 03:11:58,598 [I] Table ranger_masterkey does not exist in database rangerkms
2026-01-31 03:11:58,598 [I] --------- Importing Ranger Core DB Schema ---------
2026-01-31 03:11:58,599 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-01-31 03:11:58,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.84.20:46689/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-01-31 03:11:58,943 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260131 03:11:59.447906 19792 mini_ranger_kms.cc:310] Created kms service
I20260131 03:11:59.640039 19792 mini_ranger_kms.cc:326] Created kudu user
I20260131 03:11:59.682698 19792 mini_ranger_kms.cc:343] Created rangerkms user
Jan 31 03:11:59 dist-test-slave-5nnp krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829117, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.19.84.20@KRBTEST.COM
I20260131 03:12:00.067152 19792 mini_ranger_kms.cc:384] Added ranger policy
I20260131 03:12:00.067268 19792 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWVHG5r/build/release/bin/postgresql.jar:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWVHG5r/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260131 03:12:00.067301 19792 mini_ranger_kms.cc:240] Using host: 127.19.84.20
I20260131 03:12:00.070299 19792 mini_ranger_kms.cc:276] Ranger KMS PID: 32114
I20260131 03:12:00.070398 19792 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.19.84.20:40187
Jan 31, 2026 3:12:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jan 31, 2026 3:12:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jan 31, 2026 3:12:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jan 31, 2026 3:12:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ranger-kms/ews/webapp] .....
Jan 31, 2026 3:12:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ranger-kms/ews/webapp].
Jan 31, 2026 3:12:01 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260131 03:12:02.242370 19792 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.606s user 0.000s sys 0.183s
I20260131 03:12:02.242493 19792 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260131 03:12:02.242556 19792 mini_ranger_kms.cc:401] 127.19.84.20:40187/kms/v1/keys
Jan 31 03:12:02 dist-test-slave-5nnp krb5kdc[31696](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1769829122, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.84.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jan 31 03:12:02 dist-test-slave-5nnp krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829122, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.84.20@KRBTEST.COM for HTTP/127.19.84.20@KRBTEST.COM
Jan 31 03:12:03 dist-test-slave-5nnp krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829122, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.84.20@KRBTEST.COM for HTTP/127.19.84.20@KRBTEST.COM
Jan 31 03:12:05 dist-test-slave-5nnp krb5kdc[31696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:12:05.390113 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:43787
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:35005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:43787
--ranger_config_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.84.20:40187/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:12:05.473011 32167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:05.473235 32167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:05.473259 32167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:05.474771 32167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:12:05.474823 32167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:05.474838 32167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:05.474849 32167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:12:05.474861 32167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:12:05.476727 32167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:35005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/wal
--ranger_kms_url=127.19.84.20:40187/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:43787
--ranger_config_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:43787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:05.476967 32167 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:05.477202 32167 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:05.480146 32175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:05.480269 32167 server_base.cc:1060] running on GCE node
W20260131 03:12:05.480158 32172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:05.480224 32173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:05.480647 32167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:05.480860 32167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:05.482039 32167 hybrid_clock.cc:648] HybridClock initialized: now 1769829125482014 us; error 42 us; skew 500 ppm
Jan 31 03:12:05 dist-test-slave-5nnp krb5kdc[31696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829125, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:05.484647 32167 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:12:05.485287 32167 webserver.cc:492] Webserver started at http://127.19.84.62:41175/ using document root <none> and password file <none>
I20260131 03:12:05.485493 32167 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:05.485548 32167 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:05.485685 32167 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Jan 31 03:12:05 dist-test-slave-5nnp krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829125, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for HTTP/127.19.84.20@KRBTEST.COM
I20260131 03:12:05.539423 32167 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "6ebdf70000ba4df9a002d90e27a5b189"
format_stamp: "Formatted at 2026-01-31 03:12:05 on dist-test-slave-5nnp"
server_key: "703f55fc8dadcaa7c2a1e503c3d879cb"
server_key_iv: "578fa0840542814ba400e45e19b06beb"
server_key_version: "kuduclusterkey@0"
I20260131 03:12:05.539858 32167 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "6ebdf70000ba4df9a002d90e27a5b189"
format_stamp: "Formatted at 2026-01-31 03:12:05 on dist-test-slave-5nnp"
server_key: "703f55fc8dadcaa7c2a1e503c3d879cb"
server_key_iv: "578fa0840542814ba400e45e19b06beb"
server_key_version: "kuduclusterkey@0"
I20260131 03:12:05.541455 32167 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:12:05.553685 32182 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:05.553907 32167 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:12:05.554004 32167 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "6ebdf70000ba4df9a002d90e27a5b189"
format_stamp: "Formatted at 2026-01-31 03:12:05 on dist-test-slave-5nnp"
server_key: "703f55fc8dadcaa7c2a1e503c3d879cb"
server_key_iv: "578fa0840542814ba400e45e19b06beb"
server_key_version: "kuduclusterkey@0"
I20260131 03:12:05.554090 32167 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:05.575896 32167 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:05.577699 32167 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:05.577869 32167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:05.582288 32167 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:43787
I20260131 03:12:05.582366 32234 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:43787 every 8 connection(s)
I20260131 03:12:05.582701 32167 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:12:05.583318 32235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:12:05.585407 32235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189: Bootstrap starting.
I20260131 03:12:05.585980 32235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189: Neither blocks nor log segments found. Creating new log.
I20260131 03:12:05.586185 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 32167
I20260131 03:12:05.586252 32235 log.cc:826] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:05.586331 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:12:05.586519 19792 external_mini_cluster.cc:1468] Setting key 5a157fd6a787e08de88bcf29e9f253e1
I20260131 03:12:05.587064 32235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189: No bootstrap required, opened a new log
I20260131 03:12:05.588647 32235 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ebdf70000ba4df9a002d90e27a5b189" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 43787 } }
I20260131 03:12:05.588758 32235 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:12:05.588778 32235 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ebdf70000ba4df9a002d90e27a5b189, State: Initialized, Role: FOLLOWER
I20260131 03:12:05.588882 32235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ebdf70000ba4df9a002d90e27a5b189" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 43787 } }
I20260131 03:12:05.588941 32235 raft_consensus.cc:399] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:12:05.589000 32235 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:12:05.589046 32235 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:12:05.589826 32235 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ebdf70000ba4df9a002d90e27a5b189" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 43787 } }
I20260131 03:12:05.589937 32235 leader_election.cc:304] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ebdf70000ba4df9a002d90e27a5b189; no voters:
I20260131 03:12:05.590075 32235 leader_election.cc:290] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:12:05.590148 32240 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:12:05.590289 32235 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:12:05.590345 32240 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [term 1 LEADER]: Becoming Leader. State: Replica: 6ebdf70000ba4df9a002d90e27a5b189, State: Running, Role: LEADER
I20260131 03:12:05.590530 32240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ebdf70000ba4df9a002d90e27a5b189" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 43787 } }
I20260131 03:12:05.590835 32242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ebdf70000ba4df9a002d90e27a5b189" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ebdf70000ba4df9a002d90e27a5b189" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 43787 } } }
I20260131 03:12:05.590898 32242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [sys.catalog]: This master's current role is: LEADER
I20260131 03:12:05.591022 32242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ebdf70000ba4df9a002d90e27a5b189. Latest consensus state: current_term: 1 leader_uuid: "6ebdf70000ba4df9a002d90e27a5b189" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ebdf70000ba4df9a002d90e27a5b189" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 43787 } } }
I20260131 03:12:05.591063 32242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [sys.catalog]: This master's current role is: LEADER
I20260131 03:12:05.591260 32247 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:12:05.591765 32247 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:12:05.592049 32235 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Jan 31 03:12:05 dist-test-slave-5nnp krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:12:05.597724 32247 catalog_manager.cc:1357] Generated new cluster ID: d376fd68f4334691931570dce99de3bd
I20260131 03:12:05.597795 32247 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:12:05.599488 32239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:05.587039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60454 (local address 127.19.84.62:43787)
0131 03:12:05.587209 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:05.587221 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:05.587244 (+ 23us) server_negotiation.cc:408] Connection header received
0131 03:12:05.588053 (+ 809us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:05.588062 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:05.588118 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:05.588264 (+ 146us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:05.589281 (+ 1017us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:05.590853 (+ 1572us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:05.591696 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:05.591949 (+ 253us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:05.594882 (+ 2933us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:05.594970 (+ 88us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:05.594971 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:05.595031 (+ 60us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:05.597918 (+ 2887us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:05.598303 (+ 385us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:05.598306 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:05.598307 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:05.598368 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:05.598546 (+ 178us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:05.598547 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:05.598548 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:05.598806 (+ 258us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:05.598971 (+ 165us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:05.599044 (+ 73us) server_negotiation.cc:300] Negotiation successful
0131 03:12:05.599171 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":67,"threads_started":1}
I20260131 03:12:05.609580 32247 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:12:05.610121 32247 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:12:05.615868 32247 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189: Generated new TSK 0
I20260131 03:12:05.616226 32247 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:12:06.394212 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.84.20:40187/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:43787
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:35005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:12:06.492116 32281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:06.492378 32281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:06.492403 32281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:06.494041 32281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:06.494127 32281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:06.494172 32281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:12:06.496012 32281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:35005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.19.84.20:40187/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:43787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:06.496294 32281 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:06.496523 32281 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:06.499480 32290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:06.499480 32287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:06.500247 32288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:06.500972 32281 server_base.cc:1060] running on GCE node
I20260131 03:12:06.501160 32281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:06.501502 32281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:06.502684 32281 hybrid_clock.cc:648] HybridClock initialized: now 1769829126502652 us; error 48 us; skew 500 ppm
Jan 31 03:12:06 dist-test-slave-5nnp krb5kdc[31696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829126, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:06.505573 32281 init.cc:377] Logged in from keytab as kudu/127.19.84.1@KRBTEST.COM (short username kudu)
I20260131 03:12:06.506134 32281 webserver.cc:492] Webserver started at http://127.19.84.1:39693/ using document root <none> and password file <none>
I20260131 03:12:06.506348 32281 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:06.506404 32281 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:06.506532 32281 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Jan 31 03:12:06 dist-test-slave-5nnp krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829126, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for HTTP/127.19.84.20@KRBTEST.COM
I20260131 03:12:06.520507 32281 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "96b4a6f76d7943eb92230318e2942442"
format_stamp: "Formatted at 2026-01-31 03:12:06 on dist-test-slave-5nnp"
server_key: "61eac117a8f51120125e2bea3d643258"
server_key_iv: "33acc6995f845f9d88268213fd604d85"
server_key_version: "kuduclusterkey@0"
I20260131 03:12:06.520924 32281 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "96b4a6f76d7943eb92230318e2942442"
format_stamp: "Formatted at 2026-01-31 03:12:06 on dist-test-slave-5nnp"
server_key: "61eac117a8f51120125e2bea3d643258"
server_key_iv: "33acc6995f845f9d88268213fd604d85"
server_key_version: "kuduclusterkey@0"
I20260131 03:12:06.523451 32281 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260131 03:12:06.537917 32298 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:06.538162 32281 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:12:06.538249 32281 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "96b4a6f76d7943eb92230318e2942442"
format_stamp: "Formatted at 2026-01-31 03:12:06 on dist-test-slave-5nnp"
server_key: "61eac117a8f51120125e2bea3d643258"
server_key_iv: "33acc6995f845f9d88268213fd604d85"
server_key_version: "kuduclusterkey@0"
I20260131 03:12:06.538326 32281 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:06.560209 32281 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:06.562561 32281 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:06.562762 32281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:06.563022 32281 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:12:06.563429 32281 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:12:06.563510 32281 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:06.563567 32281 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:12:06.563611 32281 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:06.570161 32281 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:34341
I20260131 03:12:06.570621 32281 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
I20260131 03:12:06.580365 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 32281
I20260131 03:12:06.580502 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:12:06.580684 19792 external_mini_cluster.cc:1468] Setting key 4bc0eb3d82df3b0a387401c0174e1872
I20260131 03:12:06.587600 32411 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:34341 every 8 connection(s)
WARNING: no policy specified for kudu/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.2@KRBTEST.COM" created.
Jan 31 03:12:06 dist-test-slave-5nnp krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829126, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:12:06.603796 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:06.588303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:45311 (local address 127.19.84.62:43787)
0131 03:12:06.594739 (+ 6436us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:06.594742 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:06.594760 (+ 18us) server_negotiation.cc:408] Connection header received
0131 03:12:06.594794 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:06.594795 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:06.594811 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:06.594905 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:06.595924 (+ 1019us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:06.596848 (+ 924us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:06.599034 (+ 2186us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:06.599219 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:06.599991 (+ 772us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:06.599996 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:06.599997 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:06.600004 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:06.601773 (+ 1769us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:06.602231 (+ 458us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:06.602232 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:06.602233 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:06.602257 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:06.602482 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:06.602483 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:06.602484 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:06.602598 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:06.602654 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:06.603454 (+ 800us) server_negotiation.cc:300] Negotiation successful
0131 03:12:06.603649 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6397,"thread_start_us":126,"threads_started":1}
I20260131 03:12:06.603358 32415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:06.588300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:43787 (local address 127.19.84.1:45311)
0131 03:12:06.588661 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:06.588695 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:06.589729 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:06.595000 (+ 5271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:06.595005 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:06.595314 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:06.595804 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:06.595809 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:06.596949 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:06.596950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:06.597598 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:06.597601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:06.597754 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:06.598308 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:06.598335 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:06.599908 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:06.601869 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:06.601876 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:06.601879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:06.602149 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:06.602316 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:06.602317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:06.602317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:06.602421 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:06.602678 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:06.602682 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:06.602846 (+ 164us) client_negotiation.cc:769] Sending connection context
0131 03:12:06.602958 (+ 112us) client_negotiation.cc:240] Negotiation successful
0131 03:12:06.603091 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":97,"threads_started":1}
I20260131 03:12:06.604853 32412 heartbeater.cc:344] Connected to a master server at 127.19.84.62:43787
I20260131 03:12:06.604949 32412 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:06.605183 32412 heartbeater.cc:507] Master 127.19.84.62:43787 requested a full tablet report, sending...
I20260131 03:12:06.605652 32199 ts_manager.cc:194] Registered new tserver with Master: 96b4a6f76d7943eb92230318e2942442 (127.19.84.1:34341)
I20260131 03:12:06.606581 32199 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.1@KRBTEST.COM'} at 127.19.84.1:45311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:12:06.649984 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.84.20:40187/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:43787
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:35005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jan 31 03:12:06 dist-test-slave-5nnp krb5kdc[31696](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1769829126, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260131 03:12:06.767257 32422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:06.767541 32422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:06.767578 32422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:06.769964 32422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:06.770087 32422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:06.770151 32422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:12:06.772877 32422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:35005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.19.84.20:40187/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:43787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:06.773161 32422 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:06.773422 32422 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:06.776540 32431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:06.776711 32429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:06.776930 32428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:06.777130 32422 server_base.cc:1060] running on GCE node
I20260131 03:12:06.777318 32422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:06.777590 32422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:06.778759 32422 hybrid_clock.cc:648] HybridClock initialized: now 1769829126778742 us; error 37 us; skew 500 ppm
Jan 31 03:12:06 dist-test-slave-5nnp krb5kdc[31696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829126, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:06.782311 32422 init.cc:377] Logged in from keytab as kudu/127.19.84.2@KRBTEST.COM (short username kudu)
I20260131 03:12:06.783079 32422 webserver.cc:492] Webserver started at http://127.19.84.2:41889/ using document root <none> and password file <none>
I20260131 03:12:06.783315 32422 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:06.783403 32422 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:06.783579 32422 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Jan 31 03:12:06 dist-test-slave-5nnp krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829126, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for HTTP/127.19.84.20@KRBTEST.COM
I20260131 03:12:06.804786 32422 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "1152d093b3d142909fcba54ad73eeca9"
format_stamp: "Formatted at 2026-01-31 03:12:06 on dist-test-slave-5nnp"
server_key: "b5d96140235a314d882ac7515879620a"
server_key_iv: "376ed850f3d23cee27b5b6f08a6820f4"
server_key_version: "kuduclusterkey@0"
I20260131 03:12:06.805349 32422 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "1152d093b3d142909fcba54ad73eeca9"
format_stamp: "Formatted at 2026-01-31 03:12:06 on dist-test-slave-5nnp"
server_key: "b5d96140235a314d882ac7515879620a"
server_key_iv: "376ed850f3d23cee27b5b6f08a6820f4"
server_key_version: "kuduclusterkey@0"
I20260131 03:12:06.807178 32422 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:12:06.819494 32438 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:06.819741 32422 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:12:06.819818 32422 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "1152d093b3d142909fcba54ad73eeca9"
format_stamp: "Formatted at 2026-01-31 03:12:06 on dist-test-slave-5nnp"
server_key: "b5d96140235a314d882ac7515879620a"
server_key_iv: "376ed850f3d23cee27b5b6f08a6820f4"
server_key_version: "kuduclusterkey@0"
I20260131 03:12:06.819921 32422 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:06.839318 32422 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:06.841949 32422 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:06.842191 32422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:06.842552 32422 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:12:06.843000 32422 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:12:06.843040 32422 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:06.843065 32422 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:12:06.843079 32422 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:06.851050 32422 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:36649
I20260131 03:12:06.851101 32551 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:36649 every 8 connection(s)
I20260131 03:12:06.851579 32422 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
I20260131 03:12:06.856860 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 32422
I20260131 03:12:06.856986 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:12:06.857185 19792 external_mini_cluster.cc:1468] Setting key 9ff34b6a09701b67a200ed7b72534820
Jan 31 03:12:06 dist-test-slave-5nnp krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829126, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:12:06.862462 32554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:06.852246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:43787 (local address 127.19.84.2:55559)
0131 03:12:06.852471 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:06.852506 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:06.853663 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:06.854019 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:06.854023 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:06.854325 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:06.854757 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:06.854761 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:06.855873 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:06.855875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:06.856467 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:06.856470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:06.856624 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:06.857261 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:06.857288 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:06.858914 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:06.860891 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:06.860897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:06.860900 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:06.861249 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:06.861451 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:06.861452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:06.861452 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:06.861559 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:06.861824 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:06.861828 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:06.861985 (+ 157us) client_negotiation.cc:769] Sending connection context
0131 03:12:06.862101 (+ 116us) client_negotiation.cc:240] Negotiation successful
0131 03:12:06.862241 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":69,"threads_started":1}
I20260131 03:12:06.862967 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:06.852285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:55559 (local address 127.19.84.62:43787)
0131 03:12:06.852332 (+ 47us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:06.852334 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:06.852610 (+ 276us) server_negotiation.cc:408] Connection header received
0131 03:12:06.853797 (+ 1187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:06.853799 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:06.853815 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:06.853893 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:06.854879 (+ 986us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:06.855760 (+ 881us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:06.856569 (+ 809us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:06.856755 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:06.859023 (+ 2268us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:06.859030 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:06.859031 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:06.859038 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:06.860775 (+ 1737us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:06.861320 (+ 545us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:06.861322 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:06.861322 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:06.861353 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:06.861629 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:06.861630 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:06.861631 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:06.861740 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:06.861798 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:06.862782 (+ 984us) server_negotiation.cc:300] Negotiation successful
0131 03:12:06.862874 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260131 03:12:06.863719 32552 heartbeater.cc:344] Connected to a master server at 127.19.84.62:43787
I20260131 03:12:06.863831 32552 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:06.864051 32552 heartbeater.cc:507] Master 127.19.84.62:43787 requested a full tablet report, sending...
I20260131 03:12:06.864439 32199 ts_manager.cc:194] Registered new tserver with Master: 1152d093b3d142909fcba54ad73eeca9 (127.19.84.2:36649)
I20260131 03:12:06.864948 32199 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.2@KRBTEST.COM'} at 127.19.84.2:55559
WARNING: no policy specified for kudu/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:12:06.928582 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.84.20:40187/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:43787
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:35005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jan 31 03:12:07 dist-test-slave-5nnp krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829126, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for HTTP/127.19.84.20@KRBTEST.COM
W20260131 03:12:07.052466 32559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:07.052791 32559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:07.052870 32559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:07.055328 32559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:07.055454 32559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:07.055502 32559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:12:07.057540 32559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:35005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.19.84.20:40187/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:43787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:07.057811 32559 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:07.058018 32559 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:07.061056 32564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:07.061059 32567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:07.061427 32565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:07.062116 32559 server_base.cc:1060] running on GCE node
I20260131 03:12:07.062291 32559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:07.062525 32559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:07.063687 32559 hybrid_clock.cc:648] HybridClock initialized: now 1769829127063666 us; error 32 us; skew 500 ppm
Jan 31 03:12:07 dist-test-slave-5nnp krb5kdc[31696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829127, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:07.066922 32559 init.cc:377] Logged in from keytab as kudu/127.19.84.3@KRBTEST.COM (short username kudu)
I20260131 03:12:07.067687 32559 webserver.cc:492] Webserver started at http://127.19.84.3:45685/ using document root <none> and password file <none>
I20260131 03:12:07.067917 32559 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:07.067972 32559 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:07.068140 32559 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Jan 31 03:12:07 dist-test-slave-5nnp krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829127, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for HTTP/127.19.84.20@KRBTEST.COM
I20260131 03:12:07.081600 32559 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "63d1a47e7ea14d3aa18755b92ce1e90b"
format_stamp: "Formatted at 2026-01-31 03:12:07 on dist-test-slave-5nnp"
server_key: "1c0fc9b560f3d2bc48d83d560f354bcc"
server_key_iv: "8db2a3d6a6906f1fbeb87cebbc89f837"
server_key_version: "kuduclusterkey@0"
I20260131 03:12:07.082047 32559 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "63d1a47e7ea14d3aa18755b92ce1e90b"
format_stamp: "Formatted at 2026-01-31 03:12:07 on dist-test-slave-5nnp"
server_key: "1c0fc9b560f3d2bc48d83d560f354bcc"
server_key_iv: "8db2a3d6a6906f1fbeb87cebbc89f837"
server_key_version: "kuduclusterkey@0"
I20260131 03:12:07.093541 32559 fs_manager.cc:696] Time spent creating directory manager: real 0.011s user 0.001s sys 0.000s
I20260131 03:12:07.107547 32574 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:07.107806 32559 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260131 03:12:07.107918 32559 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "63d1a47e7ea14d3aa18755b92ce1e90b"
format_stamp: "Formatted at 2026-01-31 03:12:07 on dist-test-slave-5nnp"
server_key: "1c0fc9b560f3d2bc48d83d560f354bcc"
server_key_iv: "8db2a3d6a6906f1fbeb87cebbc89f837"
server_key_version: "kuduclusterkey@0"
I20260131 03:12:07.108044 32559 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:07.138849 32559 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:07.141320 32559 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:07.141464 32559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:07.141686 32559 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:12:07.142059 32559 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:12:07.142128 32559 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:07.142202 32559 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:12:07.142246 32559 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:07.149581 32559 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:33371
I20260131 03:12:07.150059 32559 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
I20260131 03:12:07.150465 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 32559
I20260131 03:12:07.150564 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:12:07.150736 19792 external_mini_cluster.cc:1468] Setting key 3625e39f4ad9f89662f2177c251f61e6
I20260131 03:12:07.159713 32687 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:33371 every 8 connection(s)
Jan 31 03:12:07 dist-test-slave-5nnp krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829127, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:12:07.170054 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:07.160048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:36043 (local address 127.19.84.62:43787)
0131 03:12:07.160093 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:07.160095 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:07.160321 (+ 226us) server_negotiation.cc:408] Connection header received
0131 03:12:07.161397 (+ 1076us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:07.161398 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:07.161412 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:07.161482 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:07.162473 (+ 991us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:07.163233 (+ 760us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:07.164087 (+ 854us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:07.164265 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:07.166807 (+ 2542us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:07.166812 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:07.166812 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:07.166820 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:07.168551 (+ 1731us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:07.169024 (+ 473us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:07.169025 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:07.169026 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:07.169048 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:07.169301 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:07.169301 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:07.169302 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:07.169418 (+ 116us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:07.169462 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:07.169841 (+ 379us) server_negotiation.cc:300] Negotiation successful
0131 03:12:07.169937 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260131 03:12:07.170791 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:07.159951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:43787 (local address 127.19.84.3:36043)
0131 03:12:07.160182 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:07.160218 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:07.161247 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:07.161608 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:07.161613 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:07.161892 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:07.162333 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:07.162338 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:07.163349 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:07.163350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:07.163972 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:07.163975 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:07.164392 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:07.165005 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:07.165030 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:07.166705 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:07.168652 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:07.168659 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:07.168663 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:07.168940 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:07.169128 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:07.169129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:07.169129 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:07.169227 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:07.169508 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:07.169514 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:07.169668 (+ 154us) client_negotiation.cc:769] Sending connection context
0131 03:12:07.170451 (+ 783us) client_negotiation.cc:240] Negotiation successful
0131 03:12:07.170585 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":81,"threads_started":1}
I20260131 03:12:07.171466 32688 heartbeater.cc:344] Connected to a master server at 127.19.84.62:43787
I20260131 03:12:07.171559 32688 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:07.171777 32688 heartbeater.cc:507] Master 127.19.84.62:43787 requested a full tablet report, sending...
I20260131 03:12:07.172174 32199 ts_manager.cc:194] Registered new tserver with Master: 63d1a47e7ea14d3aa18755b92ce1e90b (127.19.84.3:33371)
I20260131 03:12:07.172652 32199 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:36043
I20260131 03:12:07.174873 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:12:07.175637 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 32281
I20260131 03:12:07.186683 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 32422
I20260131 03:12:07.198753 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 32559
I20260131 03:12:07.207069 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 32167
I20260131 03:12:07.213553 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:43787
--webserver_interface=127.19.84.62
--webserver_port=41175
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:35005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:43787
--ranger_config_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.84.20:40187/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.84.20:40187/kms with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:12:07.296175 32691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:07.296391 32691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:07.296429 32691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:07.297914 32691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:12:07.297974 32691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:07.297997 32691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:07.298027 32691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:12:07.298050 32691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:12:07.299780 32691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:35005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.84.20:40187/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:43787
--ranger_config_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:43787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=41175
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:07.300037 32691 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:07.300225 32691 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:07.302634 32696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:07.302685 32697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:07.302780 32691 server_base.cc:1060] running on GCE node
W20260131 03:12:07.302634 32699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:07.303074 32691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:07.303318 32691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:07.304461 32691 hybrid_clock.cc:648] HybridClock initialized: now 1769829127304439 us; error 35 us; skew 500 ppm
Jan 31 03:12:07 dist-test-slave-5nnp krb5kdc[31696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829127, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:07.306730 32691 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:12:07.307235 32691 webserver.cc:492] Webserver started at http://127.19.84.62:41175/ using document root <none> and password file <none>
I20260131 03:12:07.307502 32691 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:07.307577 32691 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jan 31 03:12:07 dist-test-slave-5nnp krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829127, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for HTTP/127.19.84.20@KRBTEST.COM
I20260131 03:12:07.361621 32691 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:12:07.362490 32707 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:07.362807 32691 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:12:07.362882 32691 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "6ebdf70000ba4df9a002d90e27a5b189"
format_stamp: "Formatted at 2026-01-31 03:12:05 on dist-test-slave-5nnp"
server_key: "703f55fc8dadcaa7c2a1e503c3d879cb"
server_key_iv: "578fa0840542814ba400e45e19b06beb"
server_key_version: "kuduclusterkey@0"
I20260131 03:12:07.363207 32691 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:07.373085 32691 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:07.375499 32691 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:07.375648 32691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:07.380415 32691 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:43787
I20260131 03:12:07.380446 32759 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:43787 every 8 connection(s)
I20260131 03:12:07.380784 32691 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:12:07.381916 32760 sys_catalog.cc:263] Verifying existing consensus state
I20260131 03:12:07.382515 32760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189: Bootstrap starting.
I20260131 03:12:07.389155 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 32691
I20260131 03:12:07.389590 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.84.20:40187/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:34341
--local_ip_for_outbound_sockets=127.19.84.1
--tserver_master_addrs=127.19.84.62:43787
--webserver_port=39693
--webserver_interface=127.19.84.1
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:35005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.84.20:40187/kms with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260131 03:12:07.390537 32760 log.cc:826] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:07.392292 32760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260131 03:12:07.392599 32760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189: Bootstrap complete.
I20260131 03:12:07.394706 32760 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ebdf70000ba4df9a002d90e27a5b189" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 43787 } }
I20260131 03:12:07.394933 32760 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ebdf70000ba4df9a002d90e27a5b189, State: Initialized, Role: FOLLOWER
I20260131 03:12:07.395100 32760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ebdf70000ba4df9a002d90e27a5b189" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 43787 } }
I20260131 03:12:07.395191 32760 raft_consensus.cc:399] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:12:07.395241 32760 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:12:07.395289 32760 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [term 1 FOLLOWER]: Advancing to term 2
I20260131 03:12:07.396248 32760 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ebdf70000ba4df9a002d90e27a5b189" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 43787 } }
I20260131 03:12:07.396384 32760 leader_election.cc:304] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ebdf70000ba4df9a002d90e27a5b189; no voters:
I20260131 03:12:07.396684 32760 leader_election.cc:290] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260131 03:12:07.396713 32764 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [term 2 FOLLOWER]: Leader election won for term 2
I20260131 03:12:07.396832 32764 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [term 2 LEADER]: Becoming Leader. State: Replica: 6ebdf70000ba4df9a002d90e27a5b189, State: Running, Role: LEADER
I20260131 03:12:07.396981 32760 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:12:07.396996 32764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ebdf70000ba4df9a002d90e27a5b189" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 43787 } }
I20260131 03:12:07.397429 32765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ebdf70000ba4df9a002d90e27a5b189. Latest consensus state: current_term: 2 leader_uuid: "6ebdf70000ba4df9a002d90e27a5b189" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ebdf70000ba4df9a002d90e27a5b189" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 43787 } } }
I20260131 03:12:07.397500 32765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [sys.catalog]: This master's current role is: LEADER
I20260131 03:12:07.397760 32766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6ebdf70000ba4df9a002d90e27a5b189" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ebdf70000ba4df9a002d90e27a5b189" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 43787 } } }
I20260131 03:12:07.397854 32766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189 [sys.catalog]: This master's current role is: LEADER
I20260131 03:12:07.398178 301 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:12:07.398701 301 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:12:07.398900 301 catalog_manager.cc:1269] Loaded cluster ID: d376fd68f4334691931570dce99de3bd
I20260131 03:12:07.398985 301 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:12:07.399060 32760 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260131 03:12:07.400492 301 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:12:07.400683 301 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189: Loaded TSK: 0
I20260131 03:12:07.400902 301 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260131 03:12:07.478961 32762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:07.479348 32762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:07.479496 32762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:07.481798 32762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:07.481877 32762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:07.481925 32762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:12:07.483930 32762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:35005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.84.20:40187/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:34341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=39693
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:43787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:07.484200 32762 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:07.484419 32762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:07.487006 333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:07.487123 331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:07.487440 330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:07.487632 32762 server_base.cc:1060] running on GCE node
I20260131 03:12:07.487792 32762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:07.488031 32762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:07.489193 32762 hybrid_clock.cc:648] HybridClock initialized: now 1769829127489173 us; error 34 us; skew 500 ppm
Jan 31 03:12:07 dist-test-slave-5nnp krb5kdc[31696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829127, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:07.491824 32762 init.cc:377] Logged in from keytab as kudu/127.19.84.1@KRBTEST.COM (short username kudu)
I20260131 03:12:07.492434 32762 webserver.cc:492] Webserver started at http://127.19.84.1:39693/ using document root <none> and password file <none>
I20260131 03:12:07.492663 32762 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:07.492735 32762 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jan 31 03:12:07 dist-test-slave-5nnp krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829127, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for HTTP/127.19.84.20@KRBTEST.COM
I20260131 03:12:07.521221 32762 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260131 03:12:07.521968 341 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:07.522176 32762 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260131 03:12:07.522264 32762 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "96b4a6f76d7943eb92230318e2942442"
format_stamp: "Formatted at 2026-01-31 03:12:06 on dist-test-slave-5nnp"
server_key: "61eac117a8f51120125e2bea3d643258"
server_key_iv: "33acc6995f845f9d88268213fd604d85"
server_key_version: "kuduclusterkey@0"
I20260131 03:12:07.522617 32762 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:07.534915 32762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:07.540895 32762 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:07.541138 32762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:07.541401 32762 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:12:07.541839 32762 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:12:07.541908 32762 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:07.541962 32762 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:12:07.541998 32762 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:07.549273 32762 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:34341
I20260131 03:12:07.549304 454 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:34341 every 8 connection(s)
I20260131 03:12:07.549690 32762 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
Jan 31 03:12:07 dist-test-slave-5nnp krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829127, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:12:07.558070 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 32762
I20260131 03:12:07.558388 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.84.20:40187/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:36649
--local_ip_for_outbound_sockets=127.19.84.2
--tserver_master_addrs=127.19.84.62:43787
--webserver_port=41889
--webserver_interface=127.19.84.2
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:35005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.84.20:40187/kms with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260131 03:12:07.562644 457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:07.550603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:43787 (local address 127.19.84.1:45007)
0131 03:12:07.551034 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:07.551061 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:07.552003 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:07.552470 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:07.552474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:07.552704 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:07.552973 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:07.552976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:07.554334 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:07.554335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:07.554726 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:07.554728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:07.554863 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:07.555519 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:07.555536 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:07.556800 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:07.560811 (+ 4011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:07.560818 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:07.560822 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:07.561151 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:07.561403 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:07.561404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:07.561405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:07.561512 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:07.561943 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:07.561948 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:07.562101 (+ 153us) client_negotiation.cc:769] Sending connection context
0131 03:12:07.562218 (+ 117us) client_negotiation.cc:240] Negotiation successful
0131 03:12:07.562385 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":69,"threads_started":1}
I20260131 03:12:07.563607 458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:07.550640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:45007 (local address 127.19.84.62:43787)
0131 03:12:07.550931 (+ 291us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:07.550953 (+ 22us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:07.551183 (+ 230us) server_negotiation.cc:408] Connection header received
0131 03:12:07.552223 (+ 1040us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:07.552228 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:07.552280 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:07.552426 (+ 146us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:07.553093 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:07.554220 (+ 1127us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:07.554819 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:07.555066 (+ 247us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:07.556928 (+ 1862us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:07.556951 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:07.556952 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:07.556969 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:07.560701 (+ 3732us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:07.561253 (+ 552us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:07.561256 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:07.561257 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:07.561315 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:07.561594 (+ 279us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:07.561595 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:07.561595 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:07.561855 (+ 260us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:07.563087 (+ 1232us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:07.563178 (+ 91us) server_negotiation.cc:300] Negotiation successful
0131 03:12:07.563320 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
I20260131 03:12:07.564567 455 heartbeater.cc:344] Connected to a master server at 127.19.84.62:43787
I20260131 03:12:07.564692 455 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:07.564893 455 heartbeater.cc:507] Master 127.19.84.62:43787 requested a full tablet report, sending...
W20260131 03:12:07.565714 455 heartbeater.cc:646] Failed to heartbeat to 127.19.84.62:43787 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260131 03:12:07.565801 455 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:07.565904 455 heartbeater.cc:507] Master 127.19.84.62:43787 requested a full tablet report, sending...
I20260131 03:12:07.566296 455 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:07.566402 455 heartbeater.cc:507] Master 127.19.84.62:43787 requested a full tablet report, sending...
W20260131 03:12:07.566725 455 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260131 03:12:07.666121 459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:07.666334 459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:07.666373 459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:07.668207 459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:07.668269 459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:07.668313 459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:12:07.670064 459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:35005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.84.20:40187/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:36649
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=41889
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:43787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:07.670339 459 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:07.670564 459 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:07.673606 464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:07.673614 467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:07.674736 459 server_base.cc:1060] running on GCE node
W20260131 03:12:07.675520 465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:07.675817 459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:07.676098 459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:07.677294 459 hybrid_clock.cc:648] HybridClock initialized: now 1769829127677261 us; error 48 us; skew 500 ppm
Jan 31 03:12:07 dist-test-slave-5nnp krb5kdc[31696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829127, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:07.680505 459 init.cc:377] Logged in from keytab as kudu/127.19.84.2@KRBTEST.COM (short username kudu)
I20260131 03:12:07.681216 459 webserver.cc:492] Webserver started at http://127.19.84.2:41889/ using document root <none> and password file <none>
I20260131 03:12:07.681471 459 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:07.681543 459 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jan 31 03:12:07 dist-test-slave-5nnp krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829127, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for HTTP/127.19.84.20@KRBTEST.COM
I20260131 03:12:07.707815 459 fs_manager.cc:714] Time spent opening directory manager: real 0.009s user 0.001s sys 0.000s
I20260131 03:12:07.708488 475 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:07.708642 459 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:07.708709 459 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "1152d093b3d142909fcba54ad73eeca9"
format_stamp: "Formatted at 2026-01-31 03:12:06 on dist-test-slave-5nnp"
server_key: "b5d96140235a314d882ac7515879620a"
server_key_iv: "376ed850f3d23cee27b5b6f08a6820f4"
server_key_version: "kuduclusterkey@0"
I20260131 03:12:07.709082 459 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:07.736001 459 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:07.738327 459 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:07.738471 459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:07.738680 459 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:12:07.739025 459 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:12:07.739070 459 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:07.739117 459 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:12:07.739142 459 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:07.747057 459 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:36649
I20260131 03:12:07.747082 588 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:36649 every 8 connection(s)
I20260131 03:12:07.747560 459 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
I20260131 03:12:07.752115 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 459
I20260131 03:12:07.752480 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.84.20:40187/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:33371
--local_ip_for_outbound_sockets=127.19.84.3
--tserver_master_addrs=127.19.84.62:43787
--webserver_port=45685
--webserver_interface=127.19.84.3
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:35005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.84.20:40187/kms with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jan 31 03:12:07 dist-test-slave-5nnp krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829127, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:12:07.760142 458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:07.749636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:40929 (local address 127.19.84.62:43787)
0131 03:12:07.749736 (+ 100us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:07.749738 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:07.749749 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:12:07.750932 (+ 1183us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:07.750934 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:07.750947 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:07.751019 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:07.752010 (+ 991us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:07.752801 (+ 791us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:07.753670 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:07.753846 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:07.757168 (+ 3322us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:07.757177 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:07.757177 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:07.757193 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:07.758708 (+ 1515us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:07.759172 (+ 464us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:07.759173 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:07.759173 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:07.759201 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:07.759466 (+ 265us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:07.759467 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:07.759468 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:07.759572 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:07.759650 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:07.759936 (+ 286us) server_negotiation.cc:300] Negotiation successful
0131 03:12:07.760035 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260131 03:12:07.760457 591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:07.749421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:43787 (local address 127.19.84.2:40929)
0131 03:12:07.749618 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:07.749663 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:07.750808 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:07.751085 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:07.751089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:07.751445 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:07.751879 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:07.751884 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:07.752925 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:07.752926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:07.753551 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:07.753554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:07.753975 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:07.754604 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:07.754728 (+ 124us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:07.757022 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:07.758843 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:07.758849 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:07.758852 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:07.759089 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:07.759288 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:07.759289 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:07.759289 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:07.759410 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:07.759638 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:07.759643 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:07.759780 (+ 137us) client_negotiation.cc:769] Sending connection context
0131 03:12:07.759908 (+ 128us) client_negotiation.cc:240] Negotiation successful
0131 03:12:07.760035 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":82,"threads_started":1}
I20260131 03:12:07.761109 589 heartbeater.cc:344] Connected to a master server at 127.19.84.62:43787
I20260131 03:12:07.761222 589 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:07.761440 589 heartbeater.cc:507] Master 127.19.84.62:43787 requested a full tablet report, sending...
W20260131 03:12:07.762050 589 heartbeater.cc:646] Failed to heartbeat to 127.19.84.62:43787 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260131 03:12:07.762137 589 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:07.762240 589 heartbeater.cc:507] Master 127.19.84.62:43787 requested a full tablet report, sending...
I20260131 03:12:07.762540 589 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:07.762635 589 heartbeater.cc:507] Master 127.19.84.62:43787 requested a full tablet report, sending...
W20260131 03:12:07.762885 589 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260131 03:12:07.865576 592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:07.865852 592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:07.865897 592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:07.867851 592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:07.867924 592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:07.867967 592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:12:07.869632 592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:35005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.84.20:40187/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:33371
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=45685
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:43787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:07.869895 592 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:07.870095 592 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:07.876989 598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:07.876950 597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:07.877321 600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:07.877555 592 server_base.cc:1060] running on GCE node
I20260131 03:12:07.877771 592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:07.878006 592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:07.881290 592 hybrid_clock.cc:648] HybridClock initialized: now 1769829127881267 us; error 35 us; skew 500 ppm
Jan 31 03:12:07 dist-test-slave-5nnp krb5kdc[31696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829127, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:07.883982 592 init.cc:377] Logged in from keytab as kudu/127.19.84.3@KRBTEST.COM (short username kudu)
I20260131 03:12:07.884639 592 webserver.cc:492] Webserver started at http://127.19.84.3:45685/ using document root <none> and password file <none>
I20260131 03:12:07.884853 592 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:07.884923 592 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jan 31 03:12:07 dist-test-slave-5nnp krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829127, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for HTTP/127.19.84.20@KRBTEST.COM
I20260131 03:12:07.901376 592 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260131 03:12:07.902143 608 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:07.902354 592 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260131 03:12:07.902436 592 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "63d1a47e7ea14d3aa18755b92ce1e90b"
format_stamp: "Formatted at 2026-01-31 03:12:07 on dist-test-slave-5nnp"
server_key: "1c0fc9b560f3d2bc48d83d560f354bcc"
server_key_iv: "8db2a3d6a6906f1fbeb87cebbc89f837"
server_key_version: "kuduclusterkey@0"
I20260131 03:12:07.902798 592 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:07.923770 592 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:07.926129 592 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:07.926280 592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:07.926522 592 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:12:07.926854 592 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:12:07.926898 592 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:07.926935 592 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:12:07.926954 592 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:07.934778 592 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:33371
I20260131 03:12:07.934907 721 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:33371 every 8 connection(s)
I20260131 03:12:07.935587 592 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
I20260131 03:12:07.939087 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 592
Jan 31 03:12:07 dist-test-slave-5nnp krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829127, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:12:07.951422 458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:07.938831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:53299 (local address 127.19.84.62:43787)
0131 03:12:07.938883 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:07.938885 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:07.939495 (+ 610us) server_negotiation.cc:408] Connection header received
0131 03:12:07.940826 (+ 1331us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:07.940828 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:07.940841 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:07.940872 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:07.941896 (+ 1024us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:07.942664 (+ 768us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:07.943514 (+ 850us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:07.943714 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:07.947513 (+ 3799us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:07.947518 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:07.947519 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:07.947527 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:07.949331 (+ 1804us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:07.950040 (+ 709us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:07.950041 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:07.950042 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:07.950070 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:07.950332 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:07.950333 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:07.950333 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:07.950711 (+ 378us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:07.950761 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:07.951182 (+ 421us) server_negotiation.cc:300] Negotiation successful
0131 03:12:07.951284 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260131 03:12:07.951694 724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:07.938510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:43787 (local address 127.19.84.3:53299)
0131 03:12:07.938838 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:07.939357 (+ 519us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:07.940671 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:07.941051 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:07.941055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:07.941338 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:07.941774 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:07.941779 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:07.942782 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:07.942783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:07.943403 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:07.943406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:07.943841 (+ 435us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:07.944453 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:07.944478 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:07.946017 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:07.949654 (+ 3637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:07.949661 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:07.949664 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:07.949949 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:07.950154 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:07.950155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:07.950156 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:07.950253 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:07.950817 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:07.950822 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:07.950986 (+ 164us) client_negotiation.cc:769] Sending connection context
0131 03:12:07.951249 (+ 263us) client_negotiation.cc:240] Negotiation successful
0131 03:12:07.951416 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":71,"threads_started":1}
I20260131 03:12:07.952248 722 heartbeater.cc:344] Connected to a master server at 127.19.84.62:43787
I20260131 03:12:07.952350 722 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:07.952574 722 heartbeater.cc:507] Master 127.19.84.62:43787 requested a full tablet report, sending...
W20260131 03:12:07.953266 722 heartbeater.cc:646] Failed to heartbeat to 127.19.84.62:43787 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260131 03:12:07.953359 722 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:07.953477 722 heartbeater.cc:507] Master 127.19.84.62:43787 requested a full tablet report, sending...
I20260131 03:12:07.953848 722 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:07.953964 722 heartbeater.cc:507] Master 127.19.84.62:43787 requested a full tablet report, sending...
W20260131 03:12:07.954272 722 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260131 03:12:07.954684 726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:07.939722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60468 (local address 127.19.84.62:43787)
0131 03:12:07.943732 (+ 4010us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:07.943734 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:07.943743 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:12:07.943764 (+ 21us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:07.943765 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:07.943776 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:07.944795 (+ 1019us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:07.944848 (+ 53us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:07.945641 (+ 793us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:07.947736 (+ 2095us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:07.947943 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:07.949425 (+ 1482us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:07.949429 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:07.949429 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:07.949435 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:07.952038 (+ 2603us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:07.952855 (+ 817us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:07.952857 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:07.952857 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:07.952888 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:07.953639 (+ 751us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:07.953640 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:07.953641 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:07.953756 (+ 115us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:07.954439 (+ 683us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:07.954482 (+ 43us) server_negotiation.cc:300] Negotiation successful
0131 03:12:07.954577 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3985,"thread_start_us":50,"threads_started":1}
Jan 31 03:12:08 dist-test-slave-5nnp krb5kdc[31696](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1769829128, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:08.566919 455 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:08.567130 455 heartbeater.cc:507] Master 127.19.84.62:43787 requested a full tablet report, sending...
I20260131 03:12:08.567612 32724 ts_manager.cc:194] Registered new tserver with Master: 96b4a6f76d7943eb92230318e2942442 (127.19.84.1:34341)
I20260131 03:12:08.568667 32724 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.1@KRBTEST.COM'} at 127.19.84.1:45007
I20260131 03:12:08.763052 589 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:08.763273 589 heartbeater.cc:507] Master 127.19.84.62:43787 requested a full tablet report, sending...
I20260131 03:12:08.763684 32723 ts_manager.cc:194] Registered new tserver with Master: 1152d093b3d142909fcba54ad73eeca9 (127.19.84.2:36649)
I20260131 03:12:08.764092 32723 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.2@KRBTEST.COM'} at 127.19.84.2:40929
Jan 31 03:12:08 dist-test-slave-5nnp krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829128, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for HTTP/127.19.84.20@KRBTEST.COM
I20260131 03:12:08.954429 722 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:08.954636 722 heartbeater.cc:507] Master 127.19.84.62:43787 requested a full tablet report, sending...
I20260131 03:12:08.955020 32723 ts_manager.cc:194] Registered new tserver with Master: 63d1a47e7ea14d3aa18755b92ce1e90b (127.19.84.3:33371)
I20260131 03:12:08.955626 32723 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:53299
I20260131 03:12:08.960387 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:12:08.968628 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:08.961418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60484 (local address 127.19.84.62:43787)
0131 03:12:08.961515 (+ 97us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:08.961518 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:08.961528 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:12:08.961615 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:08.961616 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:08.961627 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:08.961696 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:08.962617 (+ 921us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:08.963470 (+ 853us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:08.964288 (+ 818us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:08.964482 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:08.965682 (+ 1200us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:08.965688 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:08.965688 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:08.965696 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:08.967555 (+ 1859us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:08.967935 (+ 380us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:08.967936 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:08.967937 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:08.967966 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:08.968157 (+ 191us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:08.968157 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:08.968158 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:08.968245 (+ 87us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:08.968328 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:08.968481 (+ 153us) server_negotiation.cc:300] Negotiation successful
0131 03:12:08.968554 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":55,"threads_started":1}
I20260131 03:12:08.970494 32723 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:12:08.971873 32723 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:12:08.982355 752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:08.974501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:34341 (local address 127.0.0.1:40542)
0131 03:12:08.974871 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:08.974904 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:08.974977 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:08.979299 (+ 4322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:08.979302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:08.979305 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:08.979623 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:08.979625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:08.980968 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:08.980969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:08.982040 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:08.982042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:08.982107 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:08.982154 (+ 47us) client_negotiation.cc:769] Sending connection context
0131 03:12:08.982202 (+ 48us) client_negotiation.cc:240] Negotiation successful
0131 03:12:08.982239 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":84,"threads_started":1}
I20260131 03:12:08.983355 753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:08.974748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40542 (local address 127.19.84.1:34341)
0131 03:12:08.979110 (+ 4362us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:08.979113 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:08.979123 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:12:08.979160 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:08.979162 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:08.979191 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:08.979271 (+ 80us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:08.979721 (+ 450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:08.980874 (+ 1153us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:08.982610 (+ 1736us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:08.983141 (+ 531us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:08.983200 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:08.983236 (+ 36us) server_negotiation.cc:300] Negotiation successful
0131 03:12:08.983278 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4328,"thread_start_us":58,"threads_started":1}
I20260131 03:12:08.983366 751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:08.974628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55008 (local address 127.19.84.2:36649)
0131 03:12:08.979027 (+ 4399us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:08.979033 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:08.979556 (+ 523us) server_negotiation.cc:408] Connection header received
0131 03:12:08.979597 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:08.979599 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:08.979617 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:08.979679 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:08.980166 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:08.981389 (+ 1223us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:08.982701 (+ 1312us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:08.983203 (+ 502us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:08.983240 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:08.983268 (+ 28us) server_negotiation.cc:300] Negotiation successful
0131 03:12:08.983300 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4354,"thread_start_us":59,"threads_started":1}
I20260131 03:12:08.983819 750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:08.974524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:36649 (local address 127.0.0.1:55008)
0131 03:12:08.979423 (+ 4899us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:08.979435 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:08.979494 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:08.979902 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:08.979903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:08.979904 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:08.980093 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:08.980093 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:08.981471 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:08.981471 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:08.982254 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:08.982255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:08.982310 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:08.982313 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:12:08.982343 (+ 30us) client_negotiation.cc:240] Negotiation successful
0131 03:12:08.982380 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4880,"thread_start_us":66,"threads_started":1}
I20260131 03:12:08.984040 389 tablet_service.cc:1505] Processing CreateTablet for tablet 814fa04f30fa42f7b3e12c611937fa17 (DEFAULT_TABLE table=test-table [id=3086646cf0744d079433417f56ba8202]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:12:08.984416 389 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 814fa04f30fa42f7b3e12c611937fa17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:12:08.984774 518 tablet_service.cc:1505] Processing CreateTablet for tablet 814fa04f30fa42f7b3e12c611937fa17 (DEFAULT_TABLE table=test-table [id=3086646cf0744d079433417f56ba8202]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:12:08.985051 518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 814fa04f30fa42f7b3e12c611937fa17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:12:08.987865 755 tablet_bootstrap.cc:492] T 814fa04f30fa42f7b3e12c611937fa17 P 96b4a6f76d7943eb92230318e2942442: Bootstrap starting.
I20260131 03:12:08.988520 755 tablet_bootstrap.cc:654] T 814fa04f30fa42f7b3e12c611937fa17 P 96b4a6f76d7943eb92230318e2942442: Neither blocks nor log segments found. Creating new log.
I20260131 03:12:08.988822 755 log.cc:826] T 814fa04f30fa42f7b3e12c611937fa17 P 96b4a6f76d7943eb92230318e2942442: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:08.989322 756 tablet_bootstrap.cc:492] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9: Bootstrap starting.
I20260131 03:12:08.990005 749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:08.974723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:33371 (local address 127.0.0.1:54536)
0131 03:12:08.979789 (+ 5066us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:08.979798 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:08.979845 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:08.987039 (+ 7194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:08.987040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:08.987043 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:08.987267 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:08.987270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:08.988858 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:08.988860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:08.989758 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:08.989760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:08.989841 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:08.989845 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:12:08.989879 (+ 34us) client_negotiation.cc:240] Negotiation successful
0131 03:12:08.989928 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5048,"thread_start_us":91,"threads_started":1}
I20260131 03:12:08.990365 756 tablet_bootstrap.cc:654] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9: Neither blocks nor log segments found. Creating new log.
I20260131 03:12:08.990692 756 log.cc:826] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:08.990988 754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:08.974936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54536 (local address 127.19.84.3:33371)
0131 03:12:08.986769 (+ 11833us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:08.986773 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:08.986789 (+ 16us) server_negotiation.cc:408] Connection header received
0131 03:12:08.986828 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:08.986831 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:08.986849 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:08.986932 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:08.987395 (+ 463us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:08.988775 (+ 1380us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:08.990215 (+ 1440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:08.990787 (+ 572us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:08.990841 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:08.990875 (+ 34us) server_negotiation.cc:300] Negotiation successful
0131 03:12:08.990914 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11792,"thread_start_us":57,"threads_started":1}
I20260131 03:12:08.991633 755 tablet_bootstrap.cc:492] T 814fa04f30fa42f7b3e12c611937fa17 P 96b4a6f76d7943eb92230318e2942442: No bootstrap required, opened a new log
I20260131 03:12:08.991755 755 ts_tablet_manager.cc:1403] T 814fa04f30fa42f7b3e12c611937fa17 P 96b4a6f76d7943eb92230318e2942442: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260131 03:12:08.991686 656 tablet_service.cc:1505] Processing CreateTablet for tablet 814fa04f30fa42f7b3e12c611937fa17 (DEFAULT_TABLE table=test-table [id=3086646cf0744d079433417f56ba8202]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:12:08.991947 656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 814fa04f30fa42f7b3e12c611937fa17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:12:08.993923 755 raft_consensus.cc:359] T 814fa04f30fa42f7b3e12c611937fa17 P 96b4a6f76d7943eb92230318e2942442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b4a6f76d7943eb92230318e2942442" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 34341 } } peers { permanent_uuid: "1152d093b3d142909fcba54ad73eeca9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 36649 } } peers { permanent_uuid: "63d1a47e7ea14d3aa18755b92ce1e90b" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 33371 } }
I20260131 03:12:08.994064 755 raft_consensus.cc:385] T 814fa04f30fa42f7b3e12c611937fa17 P 96b4a6f76d7943eb92230318e2942442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:12:08.994092 755 raft_consensus.cc:740] T 814fa04f30fa42f7b3e12c611937fa17 P 96b4a6f76d7943eb92230318e2942442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b4a6f76d7943eb92230318e2942442, State: Initialized, Role: FOLLOWER
I20260131 03:12:08.994230 755 consensus_queue.cc:260] T 814fa04f30fa42f7b3e12c611937fa17 P 96b4a6f76d7943eb92230318e2942442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b4a6f76d7943eb92230318e2942442" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 34341 } } peers { permanent_uuid: "1152d093b3d142909fcba54ad73eeca9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 36649 } } peers { permanent_uuid: "63d1a47e7ea14d3aa18755b92ce1e90b" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 33371 } }
I20260131 03:12:08.994472 755 ts_tablet_manager.cc:1434] T 814fa04f30fa42f7b3e12c611937fa17 P 96b4a6f76d7943eb92230318e2942442: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260131 03:12:08.994606 455 heartbeater.cc:499] Master 127.19.84.62:43787 was elected leader, sending a full tablet report...
I20260131 03:12:08.995244 756 tablet_bootstrap.cc:492] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9: No bootstrap required, opened a new log
I20260131 03:12:08.995361 756 ts_tablet_manager.cc:1403] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260131 03:12:08.995500 760 tablet_bootstrap.cc:492] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b: Bootstrap starting.
I20260131 03:12:08.996182 760 tablet_bootstrap.cc:654] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b: Neither blocks nor log segments found. Creating new log.
I20260131 03:12:08.996534 760 log.cc:826] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:08.997233 756 raft_consensus.cc:359] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b4a6f76d7943eb92230318e2942442" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 34341 } } peers { permanent_uuid: "1152d093b3d142909fcba54ad73eeca9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 36649 } } peers { permanent_uuid: "63d1a47e7ea14d3aa18755b92ce1e90b" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 33371 } }
I20260131 03:12:08.997362 760 tablet_bootstrap.cc:492] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b: No bootstrap required, opened a new log
I20260131 03:12:08.997377 756 raft_consensus.cc:385] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:12:08.997416 760 ts_tablet_manager.cc:1403] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:12:08.997433 756 raft_consensus.cc:740] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1152d093b3d142909fcba54ad73eeca9, State: Initialized, Role: FOLLOWER
I20260131 03:12:08.997570 756 consensus_queue.cc:260] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b4a6f76d7943eb92230318e2942442" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 34341 } } peers { permanent_uuid: "1152d093b3d142909fcba54ad73eeca9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 36649 } } peers { permanent_uuid: "63d1a47e7ea14d3aa18755b92ce1e90b" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 33371 } }
I20260131 03:12:08.997876 756 ts_tablet_manager.cc:1434] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260131 03:12:08.997895 589 heartbeater.cc:499] Master 127.19.84.62:43787 was elected leader, sending a full tablet report...
W20260131 03:12:08.998700 590 tablet.cc:2378] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:12:08.999716 760 raft_consensus.cc:359] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b4a6f76d7943eb92230318e2942442" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 34341 } } peers { permanent_uuid: "1152d093b3d142909fcba54ad73eeca9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 36649 } } peers { permanent_uuid: "63d1a47e7ea14d3aa18755b92ce1e90b" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 33371 } }
I20260131 03:12:08.999868 760 raft_consensus.cc:385] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:12:08.999902 760 raft_consensus.cc:740] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d1a47e7ea14d3aa18755b92ce1e90b, State: Initialized, Role: FOLLOWER
I20260131 03:12:08.999995 760 consensus_queue.cc:260] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b4a6f76d7943eb92230318e2942442" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 34341 } } peers { permanent_uuid: "1152d093b3d142909fcba54ad73eeca9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 36649 } } peers { permanent_uuid: "63d1a47e7ea14d3aa18755b92ce1e90b" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 33371 } }
I20260131 03:12:09.000356 760 ts_tablet_manager.cc:1434] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260131 03:12:09.000530 722 heartbeater.cc:499] Master 127.19.84.62:43787 was elected leader, sending a full tablet report...
W20260131 03:12:09.051415 456 tablet.cc:2378] T 814fa04f30fa42f7b3e12c611937fa17 P 96b4a6f76d7943eb92230318e2942442: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jan 31 03:12:09 dist-test-slave-5nnp krb5kdc[31696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829128, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for HTTP/127.19.84.20@KRBTEST.COM
I20260131 03:12:09.146564 763 raft_consensus.cc:493] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:12:09.146709 763 raft_consensus.cc:515] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b4a6f76d7943eb92230318e2942442" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 34341 } } peers { permanent_uuid: "1152d093b3d142909fcba54ad73eeca9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 36649 } } peers { permanent_uuid: "63d1a47e7ea14d3aa18755b92ce1e90b" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 33371 } }
I20260131 03:12:09.147132 763 leader_election.cc:290] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96b4a6f76d7943eb92230318e2942442 (127.19.84.1:34341), 1152d093b3d142909fcba54ad73eeca9 (127.19.84.2:36649)
I20260131 03:12:09.150058 759 raft_consensus.cc:493] T 814fa04f30fa42f7b3e12c611937fa17 P 96b4a6f76d7943eb92230318e2942442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:12:09.150192 759 raft_consensus.cc:515] T 814fa04f30fa42f7b3e12c611937fa17 P 96b4a6f76d7943eb92230318e2942442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b4a6f76d7943eb92230318e2942442" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 34341 } } peers { permanent_uuid: "1152d093b3d142909fcba54ad73eeca9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 36649 } } peers { permanent_uuid: "63d1a47e7ea14d3aa18755b92ce1e90b" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 33371 } }
I20260131 03:12:09.150542 759 leader_election.cc:290] T 814fa04f30fa42f7b3e12c611937fa17 P 96b4a6f76d7943eb92230318e2942442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1152d093b3d142909fcba54ad73eeca9 (127.19.84.2:36649), 63d1a47e7ea14d3aa18755b92ce1e90b (127.19.84.3:33371)
I20260131 03:12:09.152993 753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:09.147928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:39401 (local address 127.19.84.1:34341)
0131 03:12:09.148041 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:09.148044 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:09.148353 (+ 309us) server_negotiation.cc:408] Connection header received
0131 03:12:09.148481 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:09.148483 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:09.148494 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:09.148529 (+ 35us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:09.149029 (+ 500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.149878 (+ 849us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:09.152172 (+ 2294us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.152708 (+ 536us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:09.152720 (+ 12us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:09.152865 (+ 145us) server_negotiation.cc:300] Negotiation successful
0131 03:12:09.152911 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260131 03:12:09.153475 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:09.147365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:34341 (local address 127.19.84.3:39401)
0131 03:12:09.148274 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:09.148288 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:09.148388 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:09.148616 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:09.148617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:09.148619 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:09.148912 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:09.148916 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.150070 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:09.150071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:09.152034 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:09.152037 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.152757 (+ 720us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:09.152762 (+ 5us) client_negotiation.cc:769] Sending connection context
0131 03:12:09.153316 (+ 554us) client_negotiation.cc:240] Negotiation successful
0131 03:12:09.153370 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"thread_start_us":121,"threads_started":1}
I20260131 03:12:09.154052 762 raft_consensus.cc:493] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:12:09.154104 409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "814fa04f30fa42f7b3e12c611937fa17" candidate_uuid: "63d1a47e7ea14d3aa18755b92ce1e90b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b4a6f76d7943eb92230318e2942442" is_pre_election: true
I20260131 03:12:09.154199 409 raft_consensus.cc:2468] T 814fa04f30fa42f7b3e12c611937fa17 P 96b4a6f76d7943eb92230318e2942442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d1a47e7ea14d3aa18755b92ce1e90b in term 0.
I20260131 03:12:09.154170 762 raft_consensus.cc:515] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b4a6f76d7943eb92230318e2942442" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 34341 } } peers { permanent_uuid: "1152d093b3d142909fcba54ad73eeca9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 36649 } } peers { permanent_uuid: "63d1a47e7ea14d3aa18755b92ce1e90b" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 33371 } }
I20260131 03:12:09.154502 762 leader_election.cc:290] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96b4a6f76d7943eb92230318e2942442 (127.19.84.1:34341), 63d1a47e7ea14d3aa18755b92ce1e90b (127.19.84.3:33371)
I20260131 03:12:09.155517 610 leader_election.cc:304] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d1a47e7ea14d3aa18755b92ce1e90b, 96b4a6f76d7943eb92230318e2942442; no voters:
I20260131 03:12:09.155655 763 raft_consensus.cc:2804] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:12:09.155717 763 raft_consensus.cc:493] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:12:09.155746 763 raft_consensus.cc:3060] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:12:09.156838 763 raft_consensus.cc:515] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b4a6f76d7943eb92230318e2942442" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 34341 } } peers { permanent_uuid: "1152d093b3d142909fcba54ad73eeca9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 36649 } } peers { permanent_uuid: "63d1a47e7ea14d3aa18755b92ce1e90b" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 33371 } }
I20260131 03:12:09.156987 763 leader_election.cc:290] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b [CANDIDATE]: Term 1 election: Requested vote from peers 96b4a6f76d7943eb92230318e2942442 (127.19.84.1:34341), 1152d093b3d142909fcba54ad73eeca9 (127.19.84.2:36649)
I20260131 03:12:09.157472 409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "814fa04f30fa42f7b3e12c611937fa17" candidate_uuid: "63d1a47e7ea14d3aa18755b92ce1e90b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b4a6f76d7943eb92230318e2942442"
I20260131 03:12:09.157549 409 raft_consensus.cc:3060] T 814fa04f30fa42f7b3e12c611937fa17 P 96b4a6f76d7943eb92230318e2942442 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:12:09.158453 409 raft_consensus.cc:2468] T 814fa04f30fa42f7b3e12c611937fa17 P 96b4a6f76d7943eb92230318e2942442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d1a47e7ea14d3aa18755b92ce1e90b in term 1.
I20260131 03:12:09.158682 610 leader_election.cc:304] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d1a47e7ea14d3aa18755b92ce1e90b, 96b4a6f76d7943eb92230318e2942442; no voters:
I20260131 03:12:09.158812 763 raft_consensus.cc:2804] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:12:09.158955 763 raft_consensus.cc:697] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b [term 1 LEADER]: Becoming Leader. State: Replica: 63d1a47e7ea14d3aa18755b92ce1e90b, State: Running, Role: LEADER
I20260131 03:12:09.159051 763 consensus_queue.cc:237] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b4a6f76d7943eb92230318e2942442" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 34341 } } peers { permanent_uuid: "1152d093b3d142909fcba54ad73eeca9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 36649 } } peers { permanent_uuid: "63d1a47e7ea14d3aa18755b92ce1e90b" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 33371 } }
I20260131 03:12:09.159585 767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:09.150651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:36649 (local address 127.19.84.1:42187)
0131 03:12:09.156616 (+ 5965us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:09.156629 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:09.156695 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:09.156908 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:09.156909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:09.156911 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:09.157166 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:09.157168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.158247 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:09.158248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:09.159233 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:09.159235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.159309 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:09.159314 (+ 5us) client_negotiation.cc:769] Sending connection context
0131 03:12:09.159421 (+ 107us) client_negotiation.cc:240] Negotiation successful
0131 03:12:09.159490 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5949,"thread_start_us":113,"threads_started":1}
I20260131 03:12:09.160110 32723 catalog_manager.cc:5654] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b reported cstate change: term changed from 0 to 1, leader changed from <none> to 63d1a47e7ea14d3aa18755b92ce1e90b (127.19.84.3). New cstate: current_term: 1 leader_uuid: "63d1a47e7ea14d3aa18755b92ce1e90b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96b4a6f76d7943eb92230318e2942442" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 34341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1152d093b3d142909fcba54ad73eeca9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 36649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63d1a47e7ea14d3aa18755b92ce1e90b" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 33371 } health_report { overall_health: HEALTHY } } }
I20260131 03:12:09.160701 769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:09.150931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:42187 (local address 127.19.84.2:36649)
0131 03:12:09.156556 (+ 5625us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:09.156558 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:09.156753 (+ 195us) server_negotiation.cc:408] Connection header received
0131 03:12:09.156778 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:09.156779 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:09.156789 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:09.156851 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:09.157265 (+ 414us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.158142 (+ 877us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:09.160038 (+ 1896us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.160530 (+ 492us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:09.160540 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:09.160567 (+ 27us) server_negotiation.cc:300] Negotiation successful
0131 03:12:09.160598 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5602,"thread_start_us":94,"threads_started":1}
I20260131 03:12:09.161023 543 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "814fa04f30fa42f7b3e12c611937fa17" candidate_uuid: "96b4a6f76d7943eb92230318e2942442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1152d093b3d142909fcba54ad73eeca9" is_pre_election: true
I20260131 03:12:09.161124 543 raft_consensus.cc:2468] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96b4a6f76d7943eb92230318e2942442 in term 0.
I20260131 03:12:09.161505 342 leader_election.cc:304] T 814fa04f30fa42f7b3e12c611937fa17 P 96b4a6f76d7943eb92230318e2942442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1152d093b3d142909fcba54ad73eeca9, 96b4a6f76d7943eb92230318e2942442; no voters:
I20260131 03:12:09.161636 759 raft_consensus.cc:2764] T 814fa04f30fa42f7b3e12c611937fa17 P 96b4a6f76d7943eb92230318e2942442 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260131 03:12:09.162930 753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:09.155173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:45805 (local address 127.19.84.1:34341)
0131 03:12:09.156460 (+ 1287us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:09.156462 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:09.159607 (+ 3145us) server_negotiation.cc:408] Connection header received
0131 03:12:09.159709 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:09.159711 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:09.159720 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:09.159749 (+ 29us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:09.160330 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.160977 (+ 647us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:09.162293 (+ 1316us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.162710 (+ 417us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:09.162718 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:09.162838 (+ 120us) server_negotiation.cc:300] Negotiation successful
0131 03:12:09.162871 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1260}
I20260131 03:12:09.163160 770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:09.154719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:34341 (local address 127.19.84.2:45805)
0131 03:12:09.159537 (+ 4818us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:09.159548 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:09.159634 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:09.159855 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:09.159856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:09.159857 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:09.160098 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:09.160100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.161324 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:09.161325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:09.162193 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:09.162195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.162748 (+ 553us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:09.162752 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:12:09.163044 (+ 292us) client_negotiation.cc:240] Negotiation successful
0131 03:12:09.163084 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4800,"thread_start_us":98,"threads_started":1}
I20260131 03:12:09.163477 409 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "814fa04f30fa42f7b3e12c611937fa17" candidate_uuid: "1152d093b3d142909fcba54ad73eeca9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b4a6f76d7943eb92230318e2942442" is_pre_election: true
I20260131 03:12:09.163574 409 raft_consensus.cc:2393] T 814fa04f30fa42f7b3e12c611937fa17 P 96b4a6f76d7943eb92230318e2942442 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1152d093b3d142909fcba54ad73eeca9 in current term 1: Already voted for candidate 63d1a47e7ea14d3aa18755b92ce1e90b in this term.
I20260131 03:12:09.165025 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:09.154971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:33371 (local address 127.19.84.2:53219)
0131 03:12:09.160200 (+ 5229us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:09.160210 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:09.160257 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:09.161198 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:09.161199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:09.161201 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:09.161998 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:09.162002 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.163172 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:09.163173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:09.164788 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:09.164792 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.164870 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:09.164875 (+ 5us) client_negotiation.cc:769] Sending connection context
0131 03:12:09.164909 (+ 34us) client_negotiation.cc:240] Negotiation successful
0131 03:12:09.164947 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5211,"thread_start_us":69,"threads_started":1}
I20260131 03:12:09.165879 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:09.155215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:53219 (local address 127.19.84.3:33371)
0131 03:12:09.161085 (+ 5870us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:09.161086 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:09.161097 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:12:09.161116 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:09.161116 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:09.161126 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:09.161191 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:09.162124 (+ 933us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.162879 (+ 755us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:09.165261 (+ 2382us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.165731 (+ 470us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:09.165740 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:09.165769 (+ 29us) server_negotiation.cc:300] Negotiation successful
0131 03:12:09.165800 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5849,"thread_start_us":74,"threads_started":1}
I20260131 03:12:09.166378 676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "814fa04f30fa42f7b3e12c611937fa17" candidate_uuid: "1152d093b3d142909fcba54ad73eeca9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d1a47e7ea14d3aa18755b92ce1e90b" is_pre_election: true
I20260131 03:12:09.166754 478 leader_election.cc:304] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1152d093b3d142909fcba54ad73eeca9; no voters: 63d1a47e7ea14d3aa18755b92ce1e90b, 96b4a6f76d7943eb92230318e2942442
I20260131 03:12:09.166931 762 raft_consensus.cc:3060] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:12:09.167984 762 raft_consensus.cc:2749] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260131 03:12:09.173367 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:09.147676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:36649 (local address 127.19.84.3:35341)
0131 03:12:09.155445 (+ 7769us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:09.155458 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:09.155560 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:09.155735 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:09.155736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:09.155738 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:09.155983 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:09.155984 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.156882 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:09.156882 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:09.157783 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:09.157786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.173147 (+ 15361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:09.173155 (+ 8us) client_negotiation.cc:769] Sending connection context
0131 03:12:09.173197 (+ 42us) client_negotiation.cc:240] Negotiation successful
0131 03:12:09.173256 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7748,"thread_start_us":92,"threads_started":1}
I20260131 03:12:09.173843 751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:09.147983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:35341 (local address 127.19.84.2:36649)
0131 03:12:09.148042 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:09.148044 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:09.155531 (+ 7487us) server_negotiation.cc:408] Connection header received
0131 03:12:09.155632 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:09.155633 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:09.155643 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:09.155671 (+ 28us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:09.156064 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.156779 (+ 715us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:09.173147 (+ 16368us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.173686 (+ 539us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:09.173695 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:09.173719 (+ 24us) server_negotiation.cc:300] Negotiation successful
0131 03:12:09.173757 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260131 03:12:09.174492 543 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "814fa04f30fa42f7b3e12c611937fa17" candidate_uuid: "63d1a47e7ea14d3aa18755b92ce1e90b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1152d093b3d142909fcba54ad73eeca9" is_pre_election: true
I20260131 03:12:09.174610 543 raft_consensus.cc:2468] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63d1a47e7ea14d3aa18755b92ce1e90b in term 1.
I20260131 03:12:09.174508 542 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "814fa04f30fa42f7b3e12c611937fa17" candidate_uuid: "63d1a47e7ea14d3aa18755b92ce1e90b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1152d093b3d142909fcba54ad73eeca9"
I20260131 03:12:09.175630 542 raft_consensus.cc:2468] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d1a47e7ea14d3aa18755b92ce1e90b in term 1.
I20260131 03:12:09.176225 767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:09.150724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:33371 (local address 127.19.84.1:36723)
0131 03:12:09.159653 (+ 8929us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:09.159665 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:09.159726 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:09.173503 (+ 13777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:09.173504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:09.173511 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:09.173734 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:09.173736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.174582 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:09.174583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:09.175450 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:09.175455 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.176016 (+ 561us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:09.176021 (+ 5us) client_negotiation.cc:769] Sending connection context
0131 03:12:09.176088 (+ 67us) client_negotiation.cc:240] Negotiation successful
0131 03:12:09.176134 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8917,"thread_start_us":68,"threads_started":1}
I20260131 03:12:09.176236 754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:09.150776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:36723 (local address 127.19.84.3:33371)
0131 03:12:09.150814 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:09.150816 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:09.159714 (+ 8898us) server_negotiation.cc:408] Connection header received
0131 03:12:09.159794 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:09.159795 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:09.159804 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:09.159855 (+ 51us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:09.173830 (+ 13975us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.174472 (+ 642us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:09.175556 (+ 1084us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.175973 (+ 417us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:09.175981 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:09.176123 (+ 142us) server_negotiation.cc:300] Negotiation successful
0131 03:12:09.176169 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260131 03:12:09.176497 676 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "814fa04f30fa42f7b3e12c611937fa17" candidate_uuid: "96b4a6f76d7943eb92230318e2942442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63d1a47e7ea14d3aa18755b92ce1e90b" is_pre_election: true
W20260131 03:12:09.186820 723 tablet.cc:2378] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:12:09.210376 754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:09.207415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54550 (local address 127.19.84.3:33371)
0131 03:12:09.207465 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:09.207467 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:09.207476 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:12:09.207524 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:09.207525 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:09.207546 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:09.207611 (+ 65us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:12:09.207983 (+ 372us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.208728 (+ 745us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:09.209626 (+ 898us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:09.209798 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:09.209846 (+ 48us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:12:09.209969 (+ 123us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:12:09.210042 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:09.210263 (+ 221us) server_negotiation.cc:300] Negotiation successful
0131 03:12:09.210295 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260131 03:12:09.213083 542 raft_consensus.cc:1275] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9 [term 1 FOLLOWER]: Refusing update from remote peer 63d1a47e7ea14d3aa18755b92ce1e90b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:12:09.213423 409 raft_consensus.cc:1275] T 814fa04f30fa42f7b3e12c611937fa17 P 96b4a6f76d7943eb92230318e2942442 [term 1 FOLLOWER]: Refusing update from remote peer 63d1a47e7ea14d3aa18755b92ce1e90b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:12:09.213697 773 consensus_queue.cc:1048] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1152d093b3d142909fcba54ad73eeca9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 36649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:12:09.213835 763 consensus_queue.cc:1048] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b4a6f76d7943eb92230318e2942442" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 34341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:12:09.217496 777 mvcc.cc:204] Tried to move back new op lower bound from 7249220113250639872 to 7249220113036058624. Current Snapshot: MvccSnapshot[applied={T|T < 7249220113250639872}]
I20260131 03:12:09.221318 32723 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:12:09.221426 32723 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:12:09.222138 32723 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 6ebdf70000ba4df9a002d90e27a5b189: Sending DeleteTablet for 3 replicas of tablet 814fa04f30fa42f7b3e12c611937fa17
I20260131 03:12:09.222487 389 tablet_service.cc:1552] Processing DeleteTablet for tablet 814fa04f30fa42f7b3e12c611937fa17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:12:09 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:40542
I20260131 03:12:09.222891 518 tablet_service.cc:1552] Processing DeleteTablet for tablet 814fa04f30fa42f7b3e12c611937fa17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:12:09 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:55008
I20260131 03:12:09.223150 656 tablet_service.cc:1552] Processing DeleteTablet for tablet 814fa04f30fa42f7b3e12c611937fa17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:12:09 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:54536
I20260131 03:12:09.223760 778 mvcc.cc:204] Tried to move back new op lower bound from 7249220113250639872 to 7249220113036058624. Current Snapshot: MvccSnapshot[applied={T|T < 7249220113250639872}]
I20260131 03:12:09.225769 779 mvcc.cc:204] Tried to move back new op lower bound from 7249220113250639872 to 7249220113036058624. Current Snapshot: MvccSnapshot[applied={T|T < 7249220113250639872}]
I20260131 03:12:09.225914 784 tablet_replica.cc:333] T 814fa04f30fa42f7b3e12c611937fa17 P 96b4a6f76d7943eb92230318e2942442: stopping tablet replica
I20260131 03:12:09.225996 784 raft_consensus.cc:2243] T 814fa04f30fa42f7b3e12c611937fa17 P 96b4a6f76d7943eb92230318e2942442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:12:09.226064 784 raft_consensus.cc:2272] T 814fa04f30fa42f7b3e12c611937fa17 P 96b4a6f76d7943eb92230318e2942442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:12:09.226081 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 32762
I20260131 03:12:09.226202 785 tablet_replica.cc:333] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9: stopping tablet replica
I20260131 03:12:09.226267 785 raft_consensus.cc:2243] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:12:09.226305 785 raft_consensus.cc:2272] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:12:09.226409 786 tablet_replica.cc:333] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b: stopping tablet replica
I20260131 03:12:09.226464 786 raft_consensus.cc:2243] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:12:09.226544 786 raft_consensus.cc:2272] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:12:09.226961 786 ts_tablet_manager.cc:1916] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:12:09.232358 786 ts_tablet_manager.cc:1929] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:12:09.232868 786 log.cc:1199] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/814fa04f30fa42f7b3e12c611937fa17
I20260131 03:12:09.233327 786 ts_tablet_manager.cc:1950] T 814fa04f30fa42f7b3e12c611937fa17 P 63d1a47e7ea14d3aa18755b92ce1e90b: Deleting consensus metadata
I20260131 03:12:09.233879 32710 catalog_manager.cc:4985] TS 63d1a47e7ea14d3aa18755b92ce1e90b (127.19.84.3:33371): tablet 814fa04f30fa42f7b3e12c611937fa17 (table test-table [id=3086646cf0744d079433417f56ba8202]) successfully deleted
I20260131 03:12:09.234542 785 ts_tablet_manager.cc:1916] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:12:09.236268 785 ts_tablet_manager.cc:1929] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:12:09.236413 785 log.cc:1199] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1769828849314684-19792-0/minicluster-data/ts-1/wal/wals/814fa04f30fa42f7b3e12c611937fa17
I20260131 03:12:09.236739 785 ts_tablet_manager.cc:1950] T 814fa04f30fa42f7b3e12c611937fa17 P 1152d093b3d142909fcba54ad73eeca9: Deleting consensus metadata
I20260131 03:12:09.237293 32708 catalog_manager.cc:4985] TS 1152d093b3d142909fcba54ad73eeca9 (127.19.84.2:36649): tablet 814fa04f30fa42f7b3e12c611937fa17 (table test-table [id=3086646cf0744d079433417f56ba8202]) successfully deleted
W20260131 03:12:09.243582 32709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.84.1:34341 (error 108)
I20260131 03:12:09.244500 749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:09.244323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:34341 (local address 127.0.0.1:40554)
0131 03:12:09.244406 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:09.244461 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.1:34341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260131 03:12:09.244616 32709 catalog_manager.cc:4712] TS 96b4a6f76d7943eb92230318e2942442 (127.19.84.1:34341): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 814fa04f30fa42f7b3e12c611937fa17: Network error: Client connection negotiation failed: client connection to 127.19.84.1:34341: connect: Connection refused (error 111)
I20260131 03:12:09.247798 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 459
I20260131 03:12:09.255887 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 592
I20260131 03:12:09.262156 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 32691
I20260131 03:12:09.268863 19792 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260131 03:12:09.603873 19792 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260131 03:12:09.603979 19792 mini_ranger.cc:66] Stopping Ranger...
I20260131 03:12:09.949220 19792 mini_ranger.cc:68] Stopped Ranger
2026-01-31 03:12:09.949 UTC [31727] LOG: received smart shutdown request
2026-01-31 03:12:09.953 UTC [31727] LOG: background worker "logical replication launcher" (PID 31735) exited with exit code 1
2026-01-31 03:12:09.954 UTC [31730] LOG: shutting down
2026-01-31 03:12:09.956 UTC [31730] LOG: checkpoint starting: shutdown immediate
2026-01-31 03:12:11.062 UTC [31730] LOG: checkpoint complete: wrote 2411 buffers (14.7%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.037 s, sync=1.062 s, total=1.108 s; sync files=1108, longest=0.006 s, average=0.001 s; distance=12074 kB, estimate=12074 kB; lsn=0/2045140, redo lsn=0/2045140
2026-01-31 03:12:11.070 UTC [31727] LOG: database system is shut down
2026-01-31T03:12:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52279 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:12:11 dist-test-slave-5nnp krb5kdc[798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:12:11 dist-test-slave-5nnp krb5kdc[798](info): set up 2 sockets
Jan 31 03:12:11 dist-test-slave-5nnp krb5kdc[798](info): commencing operation
krb5kdc: starting...
W20260131 03:12:13.204326 19792 mini_kdc.cc:121] Time spent starting KDC: real 2.005s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:12:13 dist-test-slave-5nnp krb5kdc[798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:12:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:12:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:12:13.346807 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:34545
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:40607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:34545,127.19.84.61:41571,127.19.84.60:33985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:12:13.428261 814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:13.428450 814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:13.428473 814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:13.429867 814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:12:13.429909 814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:13.429924 814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:13.429936 814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:12:13.429948 814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:12:13.431576 814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:40607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:34545,127.19.84.61:41571,127.19.84.60:33985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:34545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:13.431777 814 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:13.431916 814 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:13.435252 820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:13.435346 822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:13.435245 819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:13.435422 814 server_base.cc:1060] running on GCE node
I20260131 03:12:13.435706 814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:13.435922 814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:13.437077 814 hybrid_clock.cc:648] HybridClock initialized: now 1769829133437065 us; error 26 us; skew 500 ppm
Jan 31 03:12:13 dist-test-slave-5nnp krb5kdc[798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829133, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:13.439757 814 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:12:13.440356 814 webserver.cc:492] Webserver started at http://127.19.84.62:37959/ using document root <none> and password file <none>
I20260131 03:12:13.440579 814 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:13.440636 814 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:13.440778 814 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:12:13.441774 814 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "dc85cca2574345ed971034781ea6f49e"
format_stamp: "Formatted at 2026-01-31 03:12:13 on dist-test-slave-5nnp"
server_key: "255248bd0198eaffcf35d592ec6d7107"
server_key_iv: "9f2e2b4c956c50cd9ed7fee9650bd955"
server_key_version: "encryptionkey@0"
I20260131 03:12:13.442118 814 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "dc85cca2574345ed971034781ea6f49e"
format_stamp: "Formatted at 2026-01-31 03:12:13 on dist-test-slave-5nnp"
server_key: "255248bd0198eaffcf35d592ec6d7107"
server_key_iv: "9f2e2b4c956c50cd9ed7fee9650bd955"
server_key_version: "encryptionkey@0"
I20260131 03:12:13.443552 814 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:12:13.444341 829 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:13.444541 814 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:12:13.444638 814 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "dc85cca2574345ed971034781ea6f49e"
format_stamp: "Formatted at 2026-01-31 03:12:13 on dist-test-slave-5nnp"
server_key: "255248bd0198eaffcf35d592ec6d7107"
server_key_iv: "9f2e2b4c956c50cd9ed7fee9650bd955"
server_key_version: "encryptionkey@0"
I20260131 03:12:13.444728 814 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:13.471644 814 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:13.477578 814 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:13.477766 814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:13.482478 814 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:34545
I20260131 03:12:13.482592 881 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:34545 every 8 connection(s)
I20260131 03:12:13.483096 814 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:12:13.484231 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:12:13.487234 882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34545 } has no permanent_uuid. Determining permanent_uuid...
I20260131 03:12:13.491710 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 814
I20260131 03:12:13.491858 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:12:13.492075 19792 external_mini_cluster.cc:1468] Setting key 0f7862972bb2c0d5e51fffb8c6475b2d
Jan 31 03:12:13 dist-test-slave-5nnp krb5kdc[798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829133, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:12:13.496842 884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.488046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34545 (local address 127.0.0.1:52572)
0131 03:12:13.488243 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.488273 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:13.488650 (+ 377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:13.488966 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:13.488969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:13.489291 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:13.489561 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:13.489564 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.490465 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:13.490466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:13.490823 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:13.490824 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.490928 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:13.491549 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:13.491567 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:13.493112 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:13.495508 (+ 2396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:13.495516 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:13.495519 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:13.495708 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:13.495939 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:13.495939 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:13.495940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:13.495991 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:13.496424 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:13.496431 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:13.496496 (+ 65us) client_negotiation.cc:769] Sending connection context
0131 03:12:13.496584 (+ 88us) client_negotiation.cc:240] Negotiation successful
0131 03:12:13.496703 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":104,"threads_started":1}
I20260131 03:12:13.496963 885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.488046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52572 (local address 127.19.84.62:34545)
0131 03:12:13.488777 (+ 731us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:13.488783 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:13.488793 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:12:13.488849 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:13.488851 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:13.488865 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:13.488920 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:13.489663 (+ 743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.490368 (+ 705us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:13.490928 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.491090 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:13.493205 (+ 2115us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:13.493216 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:13.493216 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:13.493225 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:13.495367 (+ 2142us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.495802 (+ 435us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.495805 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.495806 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.495865 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.496056 (+ 191us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.496057 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.496058 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.496330 (+ 272us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:13.496450 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:13.496640 (+ 190us) server_negotiation.cc:300] Negotiation successful
0131 03:12:13.496787 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":702,"thread_start_us":107,"threads_started":1}
I20260131 03:12:13.497828 882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } has no permanent_uuid. Determining permanent_uuid...
I20260131 03:12:13.498275 884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.498146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:41571 (local address 127.0.0.1:38508)
0131 03:12:13.498190 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.498242 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.61:41571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260131 03:12:13.498435 831 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.84.61:41571: connect: Connection refused (error 111)
I20260131 03:12:13.499043 884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.498935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:41571 (local address 127.0.0.1:38514)
0131 03:12:13.498987 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.499015 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.61:41571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260131 03:12:13.499154 882 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.61:41571: Network error: Client connection negotiation failed: client connection to 127.19.84.61:41571: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.19.84.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:12:13.545070 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.61:41571
--webserver_interface=127.19.84.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:40607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:34545,127.19.84.61:41571,127.19.84.60:33985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260131 03:12:13.548365 882 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } attempt: 1
I20260131 03:12:13.548830 884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.548697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:41571 (local address 127.0.0.1:38524)
0131 03:12:13.548760 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.548797 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.61:41571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260131 03:12:13.549197 884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.549070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:41571 (local address 127.0.0.1:38528)
0131 03:12:13.549115 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.549139 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.61:41571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260131 03:12:13.549286 882 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.61:41571: Network error: Client connection negotiation failed: client connection to 127.19.84.61:41571: connect: Connection refused (error 111)
I20260131 03:12:13.617440 882 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } attempt: 2
I20260131 03:12:13.617944 884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.617779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:41571 (local address 127.0.0.1:38532)
0131 03:12:13.617855 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.617903 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.61:41571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260131 03:12:13.618328 884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.618227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:41571 (local address 127.0.0.1:38538)
0131 03:12:13.618274 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.618303 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.61:41571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260131 03:12:13.618443 882 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.61:41571: Network error: Client connection negotiation failed: client connection to 127.19.84.61:41571: connect: Connection refused (error 111)
W20260131 03:12:13.626080 891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:13.626245 891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:13.626271 891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:13.627823 891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:12:13.627867 891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:13.627882 891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:13.627929 891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:12:13.627952 891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:12:13.629609 891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:40607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:34545,127.19.84.61:41571,127.19.84.60:33985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.61:41571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.84.61
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:13.629851 891 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:13.630045 891 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:13.632519 899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:13.632513 897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:13.632511 896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:13.632804 891 server_base.cc:1060] running on GCE node
I20260131 03:12:13.632987 891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:13.633198 891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:13.634331 891 hybrid_clock.cc:648] HybridClock initialized: now 1769829133634317 us; error 27 us; skew 500 ppm
Jan 31 03:12:13 dist-test-slave-5nnp krb5kdc[798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829133, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:13.636871 891 init.cc:377] Logged in from keytab as kudu/127.19.84.61@KRBTEST.COM (short username kudu)
I20260131 03:12:13.637390 891 webserver.cc:492] Webserver started at http://127.19.84.61:39185/ using document root <none> and password file <none>
I20260131 03:12:13.637598 891 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:13.637652 891 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:13.637785 891 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:12:13.638748 891 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-1/data/instance:
uuid: "1377211e100e4c7dabeb69f006e6bf66"
format_stamp: "Formatted at 2026-01-31 03:12:13 on dist-test-slave-5nnp"
server_key: "c6288e2e3551b3f5920ab0d252bc9926"
server_key_iv: "41da8d8a2b423c8787edc7d47a656b50"
server_key_version: "encryptionkey@0"
I20260131 03:12:13.639101 891 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-1/wal/instance:
uuid: "1377211e100e4c7dabeb69f006e6bf66"
format_stamp: "Formatted at 2026-01-31 03:12:13 on dist-test-slave-5nnp"
server_key: "c6288e2e3551b3f5920ab0d252bc9926"
server_key_iv: "41da8d8a2b423c8787edc7d47a656b50"
server_key_version: "encryptionkey@0"
I20260131 03:12:13.640368 891 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260131 03:12:13.641103 906 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:13.641291 891 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260131 03:12:13.641355 891 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-1/wal
uuid: "1377211e100e4c7dabeb69f006e6bf66"
format_stamp: "Formatted at 2026-01-31 03:12:13 on dist-test-slave-5nnp"
server_key: "c6288e2e3551b3f5920ab0d252bc9926"
server_key_iv: "41da8d8a2b423c8787edc7d47a656b50"
server_key_version: "encryptionkey@0"
I20260131 03:12:13.641461 891 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:13.660452 891 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:13.662441 891 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:13.662549 891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:13.666764 891 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.61:41571
I20260131 03:12:13.666851 958 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.61:41571 every 8 connection(s)
I20260131 03:12:13.667150 891 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-1/data/info.pb
I20260131 03:12:13.667879 959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:12:13.669659 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 891
I20260131 03:12:13.669751 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-1/wal/instance
I20260131 03:12:13.669924 19792 external_mini_cluster.cc:1468] Setting key ec02a4041f7b99dfb8209af87896b30c
I20260131 03:12:13.669827 959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34545 } has no permanent_uuid. Determining permanent_uuid...
Jan 31 03:12:13 dist-test-slave-5nnp krb5kdc[798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829133, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.61@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.84.60@KRBTEST.COM; defaulting to no policy
I20260131 03:12:13.678869 885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.670645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52578 (local address 127.19.84.62:34545)
0131 03:12:13.670715 (+ 70us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:13.670718 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:13.670897 (+ 179us) server_negotiation.cc:408] Connection header received
0131 03:12:13.671342 (+ 445us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:13.671343 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:13.671357 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:13.671445 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:13.672354 (+ 909us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.672818 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:13.673417 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.673611 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:13.676046 (+ 2435us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:13.676051 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:13.676052 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:13.676060 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:13.677554 (+ 1494us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.677964 (+ 410us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.677965 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.677966 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.677997 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.678200 (+ 203us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.678200 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.678201 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.678303 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:13.678381 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:13.678680 (+ 299us) server_negotiation.cc:300] Negotiation successful
0131 03:12:13.678775 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260131 03:12:13.678982 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.670569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34545 (local address 127.0.0.1:52578)
0131 03:12:13.670781 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.670811 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:13.671229 (+ 418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:13.671564 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:13.671570 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:13.671987 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:13.672265 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:13.672269 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.672940 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:13.672941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:13.673327 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:13.673328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.673468 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:13.674146 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:13.674174 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:13.675950 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:13.677652 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:13.677659 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:13.677662 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:13.677876 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:13.678062 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:13.678063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:13.678063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:13.678134 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:13.678370 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:13.678375 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:13.678536 (+ 161us) client_negotiation.cc:769] Sending connection context
0131 03:12:13.678664 (+ 128us) client_negotiation.cc:240] Negotiation successful
0131 03:12:13.678797 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":97,"threads_started":1}
I20260131 03:12:13.679616 959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.60@KRBTEST.COM" created.
Jan 31 03:12:13 dist-test-slave-5nnp krb5kdc[798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829133, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.61@KRBTEST.COM for kudu/127.19.84.61@KRBTEST.COM
I20260131 03:12:13.687264 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.679888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:41571 (local address 127.0.0.1:38544)
0131 03:12:13.679939 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.679949 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:13.679985 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:13.680526 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:13.680527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:13.680660 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:13.680799 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:13.680801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.681700 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:13.681701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:13.682293 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:13.682296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.682376 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:13.682820 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:13.682828 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:13.684254 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:13.686388 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:13.686389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:13.686389 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:13.686551 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:13.686752 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:13.686753 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:13.686754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:13.686776 (+ 22us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:13.687040 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:13.687040 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:13.687076 (+ 36us) client_negotiation.cc:769] Sending connection context
0131 03:12:13.687123 (+ 47us) client_negotiation.cc:240] Negotiation successful
0131 03:12:13.687197 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260131 03:12:13.687430 963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.680035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38544 (local address 127.19.84.61:41571)
0131 03:12:13.680373 (+ 338us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:13.680384 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:13.680400 (+ 16us) server_negotiation.cc:408] Connection header received
0131 03:12:13.680419 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:13.680421 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:13.680433 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:13.680496 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:13.680895 (+ 399us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.681586 (+ 691us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:13.682392 (+ 806us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.682661 (+ 269us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:13.684352 (+ 1691us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:13.684361 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:13.684362 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:13.684369 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:13.686291 (+ 1922us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.686638 (+ 347us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.686641 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.686642 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.686682 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.686836 (+ 154us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.686837 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.686838 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.686971 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:13.687064 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:13.687186 (+ 122us) server_negotiation.cc:300] Negotiation successful
0131 03:12:13.687277 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
I20260131 03:12:13.688104 959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } has no permanent_uuid. Determining permanent_uuid...
I20260131 03:12:13.688470 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.688342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:33985 (local address 127.0.0.1:33182)
0131 03:12:13.688388 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.688453 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260131 03:12:13.688622 910 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.84.60:33985: connect: Connection refused (error 111)
I20260131 03:12:13.688983 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.688922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:33985 (local address 127.0.0.1:33192)
0131 03:12:13.688945 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.688968 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260131 03:12:13.689113 959 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.60:33985: Network error: Client connection negotiation failed: client connection to 127.19.84.60:33985: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.60@KRBTEST.COM" created.
I20260131 03:12:13.709604 882 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } attempt: 3
Jan 31 03:12:13 dist-test-slave-5nnp krb5kdc[798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829133, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for kudu/127.19.84.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:12:13.718328 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.60:33985
--webserver_interface=127.19.84.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:40607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:34545,127.19.84.61:41571,127.19.84.60:33985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260131 03:12:13.719208 963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.710005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38558 (local address 127.19.84.61:41571)
0131 03:12:13.710075 (+ 70us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:13.710078 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:13.710088 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:12:13.710119 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:13.710121 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:13.710136 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:13.710230 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:13.710652 (+ 422us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.711111 (+ 459us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:13.711721 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.711918 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:13.713571 (+ 1653us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:13.713575 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:13.713575 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:13.713583 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:13.717961 (+ 4378us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.718416 (+ 455us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.718418 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.718418 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.718466 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.718667 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.718668 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.718669 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.718769 (+ 100us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:13.718848 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:13.719017 (+ 169us) server_negotiation.cc:300] Negotiation successful
0131 03:12:13.719103 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260131 03:12:13.719203 884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.709918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:41571 (local address 127.0.0.1:38558)
0131 03:12:13.709979 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.709991 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:13.710060 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:13.710237 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:13.710238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:13.710395 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:13.710564 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:13.710565 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.711214 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:13.711215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:13.711626 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:13.711629 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.711711 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:13.712260 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:13.712264 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:13.713487 (+ 1223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:13.718057 (+ 4570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:13.718059 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:13.718059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:13.718310 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:13.718555 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:13.718557 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:13.718557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:13.718602 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:13.718838 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:13.718839 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:13.718897 (+ 58us) client_negotiation.cc:769] Sending connection context
0131 03:12:13.718989 (+ 92us) client_negotiation.cc:240] Negotiation successful
0131 03:12:13.719104 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260131 03:12:13.719712 882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } has no permanent_uuid. Determining permanent_uuid...
I20260131 03:12:13.720091 884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.719962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:33985 (local address 127.0.0.1:33194)
0131 03:12:13.720022 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.720059 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260131 03:12:13.720521 884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.720406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:33985 (local address 127.0.0.1:33196)
0131 03:12:13.720455 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.720499 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260131 03:12:13.720633 882 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.60:33985: Network error: Client connection negotiation failed: client connection to 127.19.84.60:33985: connect: Connection refused (error 111)
I20260131 03:12:13.738328 959 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } attempt: 1
I20260131 03:12:13.738859 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.738711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:33985 (local address 127.0.0.1:33208)
0131 03:12:13.738774 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.738820 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260131 03:12:13.739240 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.739143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:33985 (local address 127.0.0.1:33220)
0131 03:12:13.739194 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.739222 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260131 03:12:13.739346 959 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.60:33985: Network error: Client connection negotiation failed: client connection to 127.19.84.60:33985: connect: Connection refused (error 111)
I20260131 03:12:13.751766 882 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } attempt: 1
I20260131 03:12:13.752265 884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.752112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:33985 (local address 127.0.0.1:33224)
0131 03:12:13.752173 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.752236 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260131 03:12:13.752636 884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.752552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:33985 (local address 127.0.0.1:33230)
0131 03:12:13.752596 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.752619 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260131 03:12:13.752758 882 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.60:33985: Network error: Client connection negotiation failed: client connection to 127.19.84.60:33985: connect: Connection refused (error 111)
W20260131 03:12:13.800715 968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:13.800918 968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:13.800966 968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:13.802462 968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:12:13.802537 968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:13.802561 968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:13.802582 968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:12:13.802603 968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:12:13.804282 968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:40607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:34545,127.19.84.61:41571,127.19.84.60:33985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.60:33985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.84.60
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:13.804560 968 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:13.804759 968 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:13.807297 974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:13.807327 976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:13.807302 973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:13.807503 959 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } attempt: 2
I20260131 03:12:13.807791 968 server_base.cc:1060] running on GCE node
I20260131 03:12:13.807900 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.807765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:33985 (local address 127.0.0.1:33242)
0131 03:12:13.807815 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.807842 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260131 03:12:13.808014 968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:13.808275 968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:13.808332 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.808251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:33985 (local address 127.0.0.1:33256)
0131 03:12:13.808285 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.808308 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260131 03:12:13.808424 959 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.60:33985: Network error: Client connection negotiation failed: client connection to 127.19.84.60:33985: connect: Connection refused (error 111)
I20260131 03:12:13.809432 968 hybrid_clock.cc:648] HybridClock initialized: now 1769829133809421 us; error 25 us; skew 500 ppm
Jan 31 03:12:13 dist-test-slave-5nnp krb5kdc[798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829133, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:13.811990 968 init.cc:377] Logged in from keytab as kudu/127.19.84.60@KRBTEST.COM (short username kudu)
I20260131 03:12:13.812500 968 webserver.cc:492] Webserver started at http://127.19.84.60:45557/ using document root <none> and password file <none>
I20260131 03:12:13.812671 968 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:13.812704 968 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:13.812827 968 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:12:13.813767 968 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-2/data/instance:
uuid: "014ec9cd45844c4395e61712c1f258d6"
format_stamp: "Formatted at 2026-01-31 03:12:13 on dist-test-slave-5nnp"
server_key: "150b7bec687b741ac32603c22e2779a9"
server_key_iv: "38fe480c129f42098f26d6058e5acec0"
server_key_version: "encryptionkey@0"
I20260131 03:12:13.814026 968 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-2/wal/instance:
uuid: "014ec9cd45844c4395e61712c1f258d6"
format_stamp: "Formatted at 2026-01-31 03:12:13 on dist-test-slave-5nnp"
server_key: "150b7bec687b741ac32603c22e2779a9"
server_key_iv: "38fe480c129f42098f26d6058e5acec0"
server_key_version: "encryptionkey@0"
I20260131 03:12:13.815238 968 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:12:13.816015 983 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:13.816269 968 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260131 03:12:13.816334 968 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-2/wal
uuid: "014ec9cd45844c4395e61712c1f258d6"
format_stamp: "Formatted at 2026-01-31 03:12:13 on dist-test-slave-5nnp"
server_key: "150b7bec687b741ac32603c22e2779a9"
server_key_iv: "38fe480c129f42098f26d6058e5acec0"
server_key_version: "encryptionkey@0"
I20260131 03:12:13.816424 968 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:13.824921 968 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:13.826715 968 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:13.826848 968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:13.827908 882 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } attempt: 2
I20260131 03:12:13.828368 884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.828219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:33985 (local address 127.0.0.1:33262)
0131 03:12:13.828287 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.828327 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260131 03:12:13.828962 884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.828831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:33985 (local address 127.0.0.1:33274)
0131 03:12:13.828890 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.828925 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260131 03:12:13.829072 882 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.60:33985: Network error: Client connection negotiation failed: client connection to 127.19.84.60:33985: connect: Connection refused (error 111)
I20260131 03:12:13.831311 968 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.60:33985
I20260131 03:12:13.831434 1035 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.60:33985 every 8 connection(s)
I20260131 03:12:13.831708 968 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-2/data/info.pb
I20260131 03:12:13.832319 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:12:13.833194 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 968
I20260131 03:12:13.833323 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACert.1769828849314684-19792-0/minicluster-data/master-2/wal/instance
I20260131 03:12:13.833496 19792 external_mini_cluster.cc:1468] Setting key 3f2151c642515e30e90c29e8040d5383
I20260131 03:12:13.834633 1036 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34545 } has no permanent_uuid. Determining permanent_uuid...
Jan 31 03:12:13 dist-test-slave-5nnp krb5kdc[798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829133, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.60@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
Jan 31 03:12:13 dist-test-slave-5nnp krb5kdc[798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.60@KRBTEST.COM
I20260131 03:12:13.842644 885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.835287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52594 (local address 127.19.84.62:34545)
0131 03:12:13.835332 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:13.835334 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:13.835515 (+ 181us) server_negotiation.cc:408] Connection header received
0131 03:12:13.835575 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:13.835575 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:13.835584 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:13.835661 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:13.836255 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.836796 (+ 541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:13.837597 (+ 801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.837784 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:13.839670 (+ 1886us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:13.839683 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:13.839683 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:13.839691 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:13.841287 (+ 1596us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.841765 (+ 478us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.841767 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.841767 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.841797 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.841992 (+ 195us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.841993 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.841994 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.842090 (+ 96us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:13.842175 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:13.842466 (+ 291us) server_negotiation.cc:300] Negotiation successful
0131 03:12:13.842552 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260131 03:12:13.842927 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.835196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34545 (local address 127.0.0.1:52594)
0131 03:12:13.835413 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.835446 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:13.835517 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:13.835693 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:13.835695 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:13.835964 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:13.836170 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:13.836173 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.836970 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:13.836971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:13.837513 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:13.837515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.837605 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:13.838132 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:13.838150 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:13.839575 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:13.841379 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:13.841386 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:13.841389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:13.841682 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:13.841861 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:13.841861 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:13.841862 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:13.841928 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:13.842164 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:13.842169 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:13.842292 (+ 123us) client_negotiation.cc:769] Sending connection context
0131 03:12:13.842434 (+ 142us) client_negotiation.cc:240] Negotiation successful
0131 03:12:13.842571 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":102,"threads_started":1}
I20260131 03:12:13.843079 1039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.834010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33278 (local address 127.19.84.60:33985)
0131 03:12:13.834143 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:13.834152 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:13.834173 (+ 21us) server_negotiation.cc:408] Connection header received
0131 03:12:13.834497 (+ 324us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:13.834502 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:13.834541 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:13.834644 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:13.835650 (+ 1006us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.836482 (+ 832us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:13.837284 (+ 802us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.837501 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:13.840048 (+ 2547us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:13.840057 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:13.840057 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:13.840064 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:13.841759 (+ 1695us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.842122 (+ 363us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.842125 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.842125 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.842164 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.842362 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.842363 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.842364 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.842467 (+ 103us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:13.842655 (+ 188us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:13.842827 (+ 172us) server_negotiation.cc:300] Negotiation successful
0131 03:12:13.842903 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260131 03:12:13.843565 1036 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } has no permanent_uuid. Determining permanent_uuid...
Jan 31 03:12:13 dist-test-slave-5nnp krb5kdc[798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829133, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.60@KRBTEST.COM for kudu/127.19.84.61@KRBTEST.COM
Jan 31 03:12:13 dist-test-slave-5nnp krb5kdc[798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:12:13.850201 963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.843894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.19.84.61:41571)
0131 03:12:13.843952 (+ 58us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:13.843954 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:13.843963 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:12:13.843987 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:13.843988 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:13.843997 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:13.844051 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:13.844598 (+ 547us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.845134 (+ 536us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:13.845670 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.845827 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:13.847462 (+ 1635us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:13.847467 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:13.847468 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:13.847476 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:13.849088 (+ 1612us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.849522 (+ 434us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.849523 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.849524 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.849545 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.849707 (+ 162us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.849708 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.849708 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.849789 (+ 81us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:13.849837 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:13.850060 (+ 223us) server_negotiation.cc:300] Negotiation successful
0131 03:12:13.850129 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260131 03:12:13.850253 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.843837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:41571 (local address 127.0.0.1:38572)
0131 03:12:13.843880 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.843890 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:13.843927 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:13.844055 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:13.844056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:13.844217 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:13.844419 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:13.844422 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.845235 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:13.845236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:13.845571 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:13.845573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.845663 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:13.846042 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:13.846046 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:13.847285 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:13.849212 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:13.849214 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:13.849214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:13.849421 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:13.849616 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:13.849616 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:13.849617 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:13.849647 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:13.849894 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:13.849895 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:13.849947 (+ 52us) client_negotiation.cc:769] Sending connection context
0131 03:12:13.850034 (+ 87us) client_negotiation.cc:240] Negotiation successful
0131 03:12:13.850125 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260131 03:12:13.850880 1036 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } has no permanent_uuid. Determining permanent_uuid...
I20260131 03:12:13.851454 885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.843854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52598 (local address 127.19.84.62:34545)
0131 03:12:13.843942 (+ 88us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:13.843943 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:13.843950 (+ 7us) server_negotiation.cc:408] Connection header received
0131 03:12:13.844206 (+ 256us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:13.844207 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:13.844214 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:13.844260 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:13.844933 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.845532 (+ 599us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:13.846230 (+ 698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.846401 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:13.848446 (+ 2045us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:13.848449 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:13.848450 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:13.848457 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:13.850127 (+ 1670us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.850595 (+ 468us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.850596 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.850596 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.850619 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.850866 (+ 247us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.850867 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.850868 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.850997 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:13.851203 (+ 206us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:13.851241 (+ 38us) server_negotiation.cc:300] Negotiation successful
0131 03:12:13.851322 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Jan 31 03:12:13 dist-test-slave-5nnp krb5kdc[798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829133, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.60@KRBTEST.COM for kudu/127.19.84.60@KRBTEST.COM
Jan 31 03:12:13 dist-test-slave-5nnp krb5kdc[798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.61@KRBTEST.COM
I20260131 03:12:13.858705 1039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.851191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33292 (local address 127.19.84.60:33985)
0131 03:12:13.851452 (+ 261us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:13.851454 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:13.851464 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:12:13.851538 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:13.851539 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:13.851551 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:13.851586 (+ 35us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:13.852246 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.852985 (+ 739us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:13.853749 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.853954 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:13.855485 (+ 1531us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:13.855488 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:13.855488 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:13.855495 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:13.857452 (+ 1957us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.857868 (+ 416us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.857870 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.857870 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.857894 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.858059 (+ 165us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.858059 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.858060 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.858169 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:13.858274 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:13.858433 (+ 159us) server_negotiation.cc:300] Negotiation successful
0131 03:12:13.858603 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":240}
I20260131 03:12:13.858819 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.851160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:33985 (local address 127.0.0.1:33292)
0131 03:12:13.851420 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.851432 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:13.851473 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:13.851652 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:13.851653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:13.851953 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:13.852138 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:13.852140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.853135 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:13.853136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:13.853648 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:13.853649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.853724 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:13.854132 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:13.854135 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:13.855316 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:13.857616 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:13.857618 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:13.857618 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:13.857792 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:13.857970 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:13.857970 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:13.857971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:13.857996 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:13.858279 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:13.858280 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:13.858325 (+ 45us) client_negotiation.cc:769] Sending connection context
0131 03:12:13.858391 (+ 66us) client_negotiation.cc:240] Negotiation successful
0131 03:12:13.858520 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244}
I20260131 03:12:13.859537 963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.852178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38582 (local address 127.19.84.61:41571)
0131 03:12:13.852245 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:13.852247 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:13.852317 (+ 70us) server_negotiation.cc:408] Connection header received
0131 03:12:13.852388 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:13.852388 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:13.852395 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:13.852431 (+ 36us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:13.853103 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.853748 (+ 645us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:13.854348 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.854561 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:13.856674 (+ 2113us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:13.856677 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:13.856678 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:13.856685 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:13.858361 (+ 1676us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.858752 (+ 391us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.858753 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.858754 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.858778 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.858971 (+ 193us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.858972 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.858972 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.859077 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:13.859241 (+ 164us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:13.859329 (+ 88us) server_negotiation.cc:300] Negotiation successful
0131 03:12:13.859443 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260131 03:12:13.860556 1036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 014ec9cd45844c4395e61712c1f258d6: Bootstrap starting.
I20260131 03:12:13.861296 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:12:13.861676 1036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 014ec9cd45844c4395e61712c1f258d6: Neither blocks nor log segments found. Creating new log.
I20260131 03:12:13.861955 1036 log.cc:826] T 00000000000000000000000000000000 P 014ec9cd45844c4395e61712c1f258d6: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:13.862785 1036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 014ec9cd45844c4395e61712c1f258d6: No bootstrap required, opened a new log
I20260131 03:12:13.864141 1036 raft_consensus.cc:359] T 00000000000000000000000000000000 P 014ec9cd45844c4395e61712c1f258d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85cca2574345ed971034781ea6f49e" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34545 } } peers { permanent_uuid: "1377211e100e4c7dabeb69f006e6bf66" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } } peers { permanent_uuid: "014ec9cd45844c4395e61712c1f258d6" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } }
I20260131 03:12:13.864274 1036 raft_consensus.cc:385] T 00000000000000000000000000000000 P 014ec9cd45844c4395e61712c1f258d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:12:13.864295 1036 raft_consensus.cc:740] T 00000000000000000000000000000000 P 014ec9cd45844c4395e61712c1f258d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014ec9cd45844c4395e61712c1f258d6, State: Initialized, Role: FOLLOWER
I20260131 03:12:13.864416 1036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 014ec9cd45844c4395e61712c1f258d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85cca2574345ed971034781ea6f49e" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34545 } } peers { permanent_uuid: "1377211e100e4c7dabeb69f006e6bf66" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } } peers { permanent_uuid: "014ec9cd45844c4395e61712c1f258d6" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } }
I20260131 03:12:13.864712 1043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 014ec9cd45844c4395e61712c1f258d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85cca2574345ed971034781ea6f49e" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34545 } } peers { permanent_uuid: "1377211e100e4c7dabeb69f006e6bf66" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } } peers { permanent_uuid: "014ec9cd45844c4395e61712c1f258d6" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } } }
I20260131 03:12:13.864722 1036 sys_catalog.cc:565] T 00000000000000000000000000000000 P 014ec9cd45844c4395e61712c1f258d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:12:13.864845 1043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 014ec9cd45844c4395e61712c1f258d6 [sys.catalog]: This master's current role is: FOLLOWER
W20260131 03:12:13.866919 1054 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 014ec9cd45844c4395e61712c1f258d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:12:13.867038 1054 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:12:13.867158 968 master_runner.cc:186] Error getting master registration for 127.19.84.62:34545: OK, instance_id { permanent_uuid: "dc85cca2574345ed971034781ea6f49e" instance_seqno: 1769829133480901 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260131 03:12:13.868877 968 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jan 31 03:12:13 dist-test-slave-5nnp krb5kdc[798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 31 03:12:13 dist-test-slave-5nnp krb5kdc[798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
Jan 31 03:12:13 dist-test-slave-5nnp krb5kdc[798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.61@KRBTEST.COM
Jan 31 03:12:13 dist-test-slave-5nnp krb5kdc[798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.60@KRBTEST.COM
I20260131 03:12:13.887929 963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.878601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38584 (local address 127.19.84.61:41571)
0131 03:12:13.878674 (+ 73us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:13.878676 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:13.878780 (+ 104us) server_negotiation.cc:408] Connection header received
0131 03:12:13.878879 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:13.878880 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:13.878890 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:13.878921 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:13.879886 (+ 965us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.880580 (+ 694us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:13.881354 (+ 774us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.881532 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:13.885054 (+ 3522us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:13.885057 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:13.885058 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:13.885065 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:13.886792 (+ 1727us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.887141 (+ 349us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.887142 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.887142 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.887164 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.887321 (+ 157us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.887321 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.887322 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.887465 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:13.887536 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:13.887749 (+ 213us) server_negotiation.cc:300] Negotiation successful
0131 03:12:13.887828 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260131 03:12:13.888177 885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.878530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52612 (local address 127.19.84.62:34545)
0131 03:12:13.878573 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:13.878576 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:13.878710 (+ 134us) server_negotiation.cc:408] Connection header received
0131 03:12:13.878825 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:13.878826 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:13.878838 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:13.878871 (+ 33us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:13.879930 (+ 1059us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.880603 (+ 673us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:13.881354 (+ 751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.881524 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:13.884656 (+ 3132us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:13.884662 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:13.884662 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:13.884670 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:13.886991 (+ 2321us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.887348 (+ 357us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.887349 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.887349 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.887404 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.887607 (+ 203us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.887608 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.887608 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.887720 (+ 112us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:13.887790 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:13.887998 (+ 208us) server_negotiation.cc:300] Negotiation successful
0131 03:12:13.888075 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260131 03:12:13.888623 1039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.878359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33294 (local address 127.19.84.60:33985)
0131 03:12:13.878407 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:13.878410 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:13.878816 (+ 406us) server_negotiation.cc:408] Connection header received
0131 03:12:13.878917 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:13.878918 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:13.878928 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:13.878958 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:13.879952 (+ 994us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.880671 (+ 719us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:13.881398 (+ 727us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.881583 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:13.885595 (+ 4012us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:13.885599 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:13.885600 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:13.885607 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:13.887499 (+ 1892us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.887812 (+ 313us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.887813 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.887813 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.887836 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.888005 (+ 169us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.888006 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.888006 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.888138 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:13.888242 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:13.888455 (+ 213us) server_negotiation.cc:300] Negotiation successful
0131 03:12:13.888540 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260131 03:12:13.899557 959 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } attempt: 3
Jan 31 03:12:13 dist-test-slave-5nnp krb5kdc[798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829133, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.61@KRBTEST.COM for kudu/127.19.84.60@KRBTEST.COM
I20260131 03:12:13.905982 1039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.899910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33310 (local address 127.19.84.60:33985)
0131 03:12:13.899972 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:13.899975 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:13.899983 (+ 8us) server_negotiation.cc:408] Connection header received
0131 03:12:13.900006 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:13.900007 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:13.900017 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:13.900082 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:13.900469 (+ 387us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.901160 (+ 691us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:13.901653 (+ 493us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.901847 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:13.903322 (+ 1475us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:13.903326 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:13.903326 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:13.903333 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:13.904971 (+ 1638us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.905330 (+ 359us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.905331 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.905331 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.905357 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.905524 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.905524 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.905525 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.905641 (+ 116us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:13.905692 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:13.905849 (+ 157us) server_negotiation.cc:300] Negotiation successful
0131 03:12:13.905917 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260131 03:12:13.906024 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.899829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:33985 (local address 127.0.0.1:33310)
0131 03:12:13.899885 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.899898 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:13.899941 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:13.900092 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:13.900092 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:13.900257 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:13.900392 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:13.900394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.901246 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:13.901246 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:13.901575 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:13.901576 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.901629 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:13.902082 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:13.902086 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:13.903237 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:13.905068 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:13.905069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:13.905069 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:13.905258 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:13.905434 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:13.905435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:13.905435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:13.905465 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:13.905704 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:13.905705 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:13.905752 (+ 47us) client_negotiation.cc:769] Sending connection context
0131 03:12:13.905825 (+ 73us) client_negotiation.cc:240] Negotiation successful
0131 03:12:13.905924 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260131 03:12:13.907523 959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66: Bootstrap starting.
I20260131 03:12:13.908469 959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66: Neither blocks nor log segments found. Creating new log.
I20260131 03:12:13.908758 959 log.cc:826] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:13.909576 959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66: No bootstrap required, opened a new log
I20260131 03:12:13.911139 959 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85cca2574345ed971034781ea6f49e" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34545 } } peers { permanent_uuid: "1377211e100e4c7dabeb69f006e6bf66" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } } peers { permanent_uuid: "014ec9cd45844c4395e61712c1f258d6" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } }
I20260131 03:12:13.911263 959 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:12:13.911294 959 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1377211e100e4c7dabeb69f006e6bf66, State: Initialized, Role: FOLLOWER
I20260131 03:12:13.911415 959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85cca2574345ed971034781ea6f49e" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34545 } } peers { permanent_uuid: "1377211e100e4c7dabeb69f006e6bf66" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } } peers { permanent_uuid: "014ec9cd45844c4395e61712c1f258d6" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } }
I20260131 03:12:13.911676 1065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85cca2574345ed971034781ea6f49e" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34545 } } peers { permanent_uuid: "1377211e100e4c7dabeb69f006e6bf66" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } } peers { permanent_uuid: "014ec9cd45844c4395e61712c1f258d6" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } } }
I20260131 03:12:13.911718 959 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:12:13.911777 1065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [sys.catalog]: This master's current role is: FOLLOWER
W20260131 03:12:13.913296 912 tablet.cc:2378] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260131 03:12:13.913724 1076 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:12:13.913784 1076 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:12:13.913843 891 master_runner.cc:186] Error getting master registration for 127.19.84.62:34545: OK, instance_id { permanent_uuid: "dc85cca2574345ed971034781ea6f49e" instance_seqno: 1769829133480901 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260131 03:12:13.916000 891 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260131 03:12:13.928236 882 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } attempt: 3
Jan 31 03:12:13 dist-test-slave-5nnp krb5kdc[798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829133, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for kudu/127.19.84.60@KRBTEST.COM
I20260131 03:12:13.936090 884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.928561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:33985 (local address 127.0.0.1:33324)
0131 03:12:13.928621 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:13.928634 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:13.928674 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:13.928828 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:13.928829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:13.929050 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:13.929259 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:13.929261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.930177 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:13.930179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:13.930745 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:13.930747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.930831 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:13.931486 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:13.931491 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:13.933051 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:13.935039 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:13.935040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:13.935041 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:13.935244 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:13.935457 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:13.935458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:13.935459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:13.935494 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:13.935752 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:13.935753 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:13.935806 (+ 53us) client_negotiation.cc:769] Sending connection context
0131 03:12:13.935879 (+ 73us) client_negotiation.cc:240] Negotiation successful
0131 03:12:13.935982 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260131 03:12:13.936100 1039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:13.928629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33324 (local address 127.19.84.60:33985)
0131 03:12:13.928685 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:13.928688 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:13.928698 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:12:13.928726 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:13.928727 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:13.928737 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:13.928807 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:13.929360 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.930067 (+ 707us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:13.930834 (+ 767us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:13.931012 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:13.933166 (+ 2154us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:13.933170 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:13.933170 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:13.933179 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:13.934942 (+ 1763us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.935326 (+ 384us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.935327 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.935327 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.935355 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:13.935561 (+ 206us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:13.935562 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:13.935562 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:13.935682 (+ 120us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:13.935753 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:13.935919 (+ 166us) server_negotiation.cc:300] Negotiation successful
0131 03:12:13.936003 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260131 03:12:13.937973 882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc85cca2574345ed971034781ea6f49e: Bootstrap starting.
I20260131 03:12:13.938891 882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc85cca2574345ed971034781ea6f49e: Neither blocks nor log segments found. Creating new log.
I20260131 03:12:13.939177 882 log.cc:826] T 00000000000000000000000000000000 P dc85cca2574345ed971034781ea6f49e: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:13.939967 882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc85cca2574345ed971034781ea6f49e: No bootstrap required, opened a new log
I20260131 03:12:13.941567 882 raft_consensus.cc:359] T 00000000000000000000000000000000 P dc85cca2574345ed971034781ea6f49e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85cca2574345ed971034781ea6f49e" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34545 } } peers { permanent_uuid: "1377211e100e4c7dabeb69f006e6bf66" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } } peers { permanent_uuid: "014ec9cd45844c4395e61712c1f258d6" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } }
I20260131 03:12:13.941694 882 raft_consensus.cc:385] T 00000000000000000000000000000000 P dc85cca2574345ed971034781ea6f49e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:12:13.941725 882 raft_consensus.cc:740] T 00000000000000000000000000000000 P dc85cca2574345ed971034781ea6f49e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc85cca2574345ed971034781ea6f49e, State: Initialized, Role: FOLLOWER
I20260131 03:12:13.941829 882 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc85cca2574345ed971034781ea6f49e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85cca2574345ed971034781ea6f49e" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34545 } } peers { permanent_uuid: "1377211e100e4c7dabeb69f006e6bf66" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } } peers { permanent_uuid: "014ec9cd45844c4395e61712c1f258d6" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } }
I20260131 03:12:13.942126 882 sys_catalog.cc:565] T 00000000000000000000000000000000 P dc85cca2574345ed971034781ea6f49e [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:12:13.942086 1079 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc85cca2574345ed971034781ea6f49e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85cca2574345ed971034781ea6f49e" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34545 } } peers { permanent_uuid: "1377211e100e4c7dabeb69f006e6bf66" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } } peers { permanent_uuid: "014ec9cd45844c4395e61712c1f258d6" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } } }
I20260131 03:12:13.942287 1079 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc85cca2574345ed971034781ea6f49e [sys.catalog]: This master's current role is: FOLLOWER
W20260131 03:12:13.944063 1090 catalog_manager.cc:1568] T 00000000000000000000000000000000 P dc85cca2574345ed971034781ea6f49e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:12:13.944181 1090 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:12:13.946410 814 master_runner.cc:263] No leader master found from master dc85cca2574345ed971034781ea6f49e
I20260131 03:12:13.946465 814 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260131 03:12:13.964510 1065 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:12:13.964668 1065 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85cca2574345ed971034781ea6f49e" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34545 } } peers { permanent_uuid: "1377211e100e4c7dabeb69f006e6bf66" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } } peers { permanent_uuid: "014ec9cd45844c4395e61712c1f258d6" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } }
I20260131 03:12:13.964946 1065 leader_election.cc:290] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc85cca2574345ed971034781ea6f49e (127.19.84.62:34545), 014ec9cd45844c4395e61712c1f258d6 (127.19.84.60:33985)
I20260131 03:12:13.965222 1010 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1377211e100e4c7dabeb69f006e6bf66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014ec9cd45844c4395e61712c1f258d6" is_pre_election: true
I20260131 03:12:13.965374 1010 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 014ec9cd45844c4395e61712c1f258d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1377211e100e4c7dabeb69f006e6bf66 in term 0.
I20260131 03:12:13.965433 856 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1377211e100e4c7dabeb69f006e6bf66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc85cca2574345ed971034781ea6f49e" is_pre_election: true
I20260131 03:12:13.965564 856 raft_consensus.cc:2468] T 00000000000000000000000000000000 P dc85cca2574345ed971034781ea6f49e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1377211e100e4c7dabeb69f006e6bf66 in term 0.
I20260131 03:12:13.965549 910 leader_election.cc:304] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014ec9cd45844c4395e61712c1f258d6, 1377211e100e4c7dabeb69f006e6bf66; no voters:
I20260131 03:12:13.965662 1065 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:12:13.965711 1065 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:12:13.965734 1065 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:12:13.966694 1065 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85cca2574345ed971034781ea6f49e" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34545 } } peers { permanent_uuid: "1377211e100e4c7dabeb69f006e6bf66" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } } peers { permanent_uuid: "014ec9cd45844c4395e61712c1f258d6" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } }
I20260131 03:12:13.966825 1065 leader_election.cc:290] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [CANDIDATE]: Term 1 election: Requested vote from peers dc85cca2574345ed971034781ea6f49e (127.19.84.62:34545), 014ec9cd45844c4395e61712c1f258d6 (127.19.84.60:33985)
I20260131 03:12:13.967013 856 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1377211e100e4c7dabeb69f006e6bf66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc85cca2574345ed971034781ea6f49e"
I20260131 03:12:13.967097 856 raft_consensus.cc:3060] T 00000000000000000000000000000000 P dc85cca2574345ed971034781ea6f49e [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:12:13.968011 856 raft_consensus.cc:2468] T 00000000000000000000000000000000 P dc85cca2574345ed971034781ea6f49e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1377211e100e4c7dabeb69f006e6bf66 in term 1.
I20260131 03:12:13.968189 907 leader_election.cc:304] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1377211e100e4c7dabeb69f006e6bf66, dc85cca2574345ed971034781ea6f49e; no voters:
I20260131 03:12:13.968395 1010 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1377211e100e4c7dabeb69f006e6bf66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014ec9cd45844c4395e61712c1f258d6"
I20260131 03:12:13.968468 1010 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 014ec9cd45844c4395e61712c1f258d6 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:12:13.969187 1010 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 014ec9cd45844c4395e61712c1f258d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1377211e100e4c7dabeb69f006e6bf66 in term 1.
I20260131 03:12:13.969388 1065 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:12:13.969545 1065 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [term 1 LEADER]: Becoming Leader. State: Replica: 1377211e100e4c7dabeb69f006e6bf66, State: Running, Role: LEADER
I20260131 03:12:13.969632 1065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85cca2574345ed971034781ea6f49e" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34545 } } peers { permanent_uuid: "1377211e100e4c7dabeb69f006e6bf66" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } } peers { permanent_uuid: "014ec9cd45844c4395e61712c1f258d6" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } }
I20260131 03:12:13.969954 1065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1377211e100e4c7dabeb69f006e6bf66. Latest consensus state: current_term: 1 leader_uuid: "1377211e100e4c7dabeb69f006e6bf66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85cca2574345ed971034781ea6f49e" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34545 } } peers { permanent_uuid: "1377211e100e4c7dabeb69f006e6bf66" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } } peers { permanent_uuid: "014ec9cd45844c4395e61712c1f258d6" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } } }
I20260131 03:12:13.970019 1065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [sys.catalog]: This master's current role is: LEADER
I20260131 03:12:13.970667 1094 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:12:13.970849 1094 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:12:13.972208 1010 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 014ec9cd45844c4395e61712c1f258d6 [term 1 FOLLOWER]: Refusing update from remote peer 1377211e100e4c7dabeb69f006e6bf66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:12:13.972447 1092 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "014ec9cd45844c4395e61712c1f258d6" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:12:13.973440 1043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 014ec9cd45844c4395e61712c1f258d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1377211e100e4c7dabeb69f006e6bf66. Latest consensus state: current_term: 1 leader_uuid: "1377211e100e4c7dabeb69f006e6bf66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85cca2574345ed971034781ea6f49e" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34545 } } peers { permanent_uuid: "1377211e100e4c7dabeb69f006e6bf66" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } } peers { permanent_uuid: "014ec9cd45844c4395e61712c1f258d6" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } } }
I20260131 03:12:13.973519 1043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 014ec9cd45844c4395e61712c1f258d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:12:13.973814 856 raft_consensus.cc:1275] T 00000000000000000000000000000000 P dc85cca2574345ed971034781ea6f49e [term 1 FOLLOWER]: Refusing update from remote peer 1377211e100e4c7dabeb69f006e6bf66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:12:13.974020 1065 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc85cca2574345ed971034781ea6f49e" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:12:13.974931 1079 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc85cca2574345ed971034781ea6f49e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1377211e100e4c7dabeb69f006e6bf66. Latest consensus state: current_term: 1 leader_uuid: "1377211e100e4c7dabeb69f006e6bf66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85cca2574345ed971034781ea6f49e" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34545 } } peers { permanent_uuid: "1377211e100e4c7dabeb69f006e6bf66" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } } peers { permanent_uuid: "014ec9cd45844c4395e61712c1f258d6" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } } }
I20260131 03:12:13.975010 1079 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc85cca2574345ed971034781ea6f49e [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:12:13.976100 1092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1377211e100e4c7dabeb69f006e6bf66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85cca2574345ed971034781ea6f49e" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34545 } } peers { permanent_uuid: "1377211e100e4c7dabeb69f006e6bf66" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } } peers { permanent_uuid: "014ec9cd45844c4395e61712c1f258d6" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } } }
I20260131 03:12:13.976176 1092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [sys.catalog]: This master's current role is: LEADER
I20260131 03:12:13.976593 1079 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc85cca2574345ed971034781ea6f49e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1377211e100e4c7dabeb69f006e6bf66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85cca2574345ed971034781ea6f49e" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34545 } } peers { permanent_uuid: "1377211e100e4c7dabeb69f006e6bf66" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } } peers { permanent_uuid: "014ec9cd45844c4395e61712c1f258d6" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } } }
I20260131 03:12:13.976665 1079 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc85cca2574345ed971034781ea6f49e [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:12:13.976814 1095 mvcc.cc:204] Tried to move back new op lower bound from 7249220132748111872 to 7249220132740071424. Current Snapshot: MvccSnapshot[applied={T|T < 7249220132748111872}]
I20260131 03:12:13.977073 1098 mvcc.cc:204] Tried to move back new op lower bound from 7249220132748111872 to 7249220132740071424. Current Snapshot: MvccSnapshot[applied={T|T < 7249220132748111872}]
I20260131 03:12:13.977984 1065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1377211e100e4c7dabeb69f006e6bf66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85cca2574345ed971034781ea6f49e" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34545 } } peers { permanent_uuid: "1377211e100e4c7dabeb69f006e6bf66" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } } peers { permanent_uuid: "014ec9cd45844c4395e61712c1f258d6" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } } }
I20260131 03:12:13.978080 1065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66 [sys.catalog]: This master's current role is: LEADER
I20260131 03:12:13.978457 1043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 014ec9cd45844c4395e61712c1f258d6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1377211e100e4c7dabeb69f006e6bf66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc85cca2574345ed971034781ea6f49e" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34545 } } peers { permanent_uuid: "1377211e100e4c7dabeb69f006e6bf66" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 41571 } } peers { permanent_uuid: "014ec9cd45844c4395e61712c1f258d6" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 33985 } } }
I20260131 03:12:13.978538 1043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 014ec9cd45844c4395e61712c1f258d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:12:13.978715 1096 mvcc.cc:204] Tried to move back new op lower bound from 7249220132748111872 to 7249220132740071424. Current Snapshot: MvccSnapshot[applied={T|T < 7249220132748111872}]
I20260131 03:12:13.979223 1094 catalog_manager.cc:1357] Generated new cluster ID: 262e27475e75458c98458983868dcf14
I20260131 03:12:13.979269 1094 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:12:13.997773 1094 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:12:13.998533 1094 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:12:14.012010 1094 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 1377211e100e4c7dabeb69f006e6bf66: Generated new TSK 0
I20260131 03:12:14.012233 1094 catalog_manager.cc:1524] Initializing in-progress tserver states...
Jan 31 03:12:14 dist-test-slave-5nnp krb5kdc[798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.19.84.61@KRBTEST.COM
Jan 31 03:12:14 dist-test-slave-5nnp krb5kdc[798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.19.84.62@KRBTEST.COM
Jan 31 03:12:14 dist-test-slave-5nnp krb5kdc[798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.19.84.60@KRBTEST.COM
I20260131 03:12:14.867527 1054 catalog_manager.cc:1269] Loaded cluster ID: 262e27475e75458c98458983868dcf14
I20260131 03:12:14.867610 1054 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 014ec9cd45844c4395e61712c1f258d6: loading cluster ID for follower catalog manager: success
I20260131 03:12:14.868531 1054 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 014ec9cd45844c4395e61712c1f258d6: acquiring CA information for follower catalog manager: success
I20260131 03:12:14.868858 1054 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 014ec9cd45844c4395e61712c1f258d6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260131 03:12:14.944669 1090 catalog_manager.cc:1269] Loaded cluster ID: 262e27475e75458c98458983868dcf14
I20260131 03:12:14.944749 1090 catalog_manager.cc:1562] T 00000000000000000000000000000000 P dc85cca2574345ed971034781ea6f49e: loading cluster ID for follower catalog manager: success
I20260131 03:12:14.945758 1090 catalog_manager.cc:1584] T 00000000000000000000000000000000 P dc85cca2574345ed971034781ea6f49e: acquiring CA information for follower catalog manager: success
I20260131 03:12:14.946076 1090 catalog_manager.cc:1612] T 00000000000000000000000000000000 P dc85cca2574345ed971034781ea6f49e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260131 03:12:15.148805 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 814
I20260131 03:12:15.154309 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 891
I20260131 03:12:15.159317 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 968
2026-01-31T03:12:15Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3980 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:12:15 dist-test-slave-5nnp krb5kdc[1110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:12:15 dist-test-slave-5nnp krb5kdc[1110](info): set up 2 sockets
Jan 31 03:12:15 dist-test-slave-5nnp krb5kdc[1110](info): commencing operation
krb5kdc: starting...
W20260131 03:12:17.167913 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.988s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:12:17 dist-test-slave-5nnp krb5kdc[1110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:12:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:12:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:12:17.308126 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:42505
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:37999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:42505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:12:17.386225 1126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:17.386415 1126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:17.386443 1126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:17.387952 1126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:12:17.388000 1126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:17.388015 1126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:17.388028 1126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:12:17.388041 1126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:12:17.389616 1126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:37999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:42505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:42505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:17.389833 1126 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:17.389967 1126 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260131 03:12:17.392701 1126 server_base.cc:1060] running on GCE node
W20260131 03:12:17.392676 1131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:17.392839 1134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:17.392691 1132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:17.393245 1126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:17.393510 1126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:17.394663 1126 hybrid_clock.cc:648] HybridClock initialized: now 1769829137394660 us; error 36 us; skew 500 ppm
Jan 31 03:12:17 dist-test-slave-5nnp krb5kdc[1110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829137, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:17.397071 1126 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:12:17.397578 1126 webserver.cc:492] Webserver started at http://127.19.84.62:46681/ using document root <none> and password file <none>
I20260131 03:12:17.397744 1126 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:17.397776 1126 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:17.397855 1126 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:12:17.398746 1126 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "4aaa42d3a652473db5956a92437c08e3"
format_stamp: "Formatted at 2026-01-31 03:12:17 on dist-test-slave-5nnp"
server_key: "8651b9d16b992ca5939d7a1425404e09"
server_key_iv: "f25c533057247a4ad1ccbfad1a648cc9"
server_key_version: "encryptionkey@0"
I20260131 03:12:17.399003 1126 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "4aaa42d3a652473db5956a92437c08e3"
format_stamp: "Formatted at 2026-01-31 03:12:17 on dist-test-slave-5nnp"
server_key: "8651b9d16b992ca5939d7a1425404e09"
server_key_iv: "f25c533057247a4ad1ccbfad1a648cc9"
server_key_version: "encryptionkey@0"
I20260131 03:12:17.400233 1126 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260131 03:12:17.400899 1141 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:17.401082 1126 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260131 03:12:17.401153 1126 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "4aaa42d3a652473db5956a92437c08e3"
format_stamp: "Formatted at 2026-01-31 03:12:17 on dist-test-slave-5nnp"
server_key: "8651b9d16b992ca5939d7a1425404e09"
server_key_iv: "f25c533057247a4ad1ccbfad1a648cc9"
server_key_version: "encryptionkey@0"
I20260131 03:12:17.401202 1126 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:17.409176 1126 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:17.411065 1126 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:17.411167 1126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:17.415052 1126 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:42505
I20260131 03:12:17.415123 1193 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:42505 every 8 connection(s)
I20260131 03:12:17.415508 1126 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:12:17.416164 1194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:12:17.418638 1194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aaa42d3a652473db5956a92437c08e3: Bootstrap starting.
I20260131 03:12:17.419224 1194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4aaa42d3a652473db5956a92437c08e3: Neither blocks nor log segments found. Creating new log.
I20260131 03:12:17.419589 1194 log.cc:826] T 00000000000000000000000000000000 P 4aaa42d3a652473db5956a92437c08e3: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:17.420359 1194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aaa42d3a652473db5956a92437c08e3: No bootstrap required, opened a new log
I20260131 03:12:17.421551 1194 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4aaa42d3a652473db5956a92437c08e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aaa42d3a652473db5956a92437c08e3" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42505 } }
I20260131 03:12:17.421694 1194 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4aaa42d3a652473db5956a92437c08e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:12:17.421737 1194 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4aaa42d3a652473db5956a92437c08e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aaa42d3a652473db5956a92437c08e3, State: Initialized, Role: FOLLOWER
I20260131 03:12:17.421875 1194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aaa42d3a652473db5956a92437c08e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aaa42d3a652473db5956a92437c08e3" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42505 } }
I20260131 03:12:17.421954 1194 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4aaa42d3a652473db5956a92437c08e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:12:17.421990 1194 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4aaa42d3a652473db5956a92437c08e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:12:17.422037 1194 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4aaa42d3a652473db5956a92437c08e3 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:12:17.422495 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 1126
I20260131 03:12:17.422583 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:12:17.422574 1194 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4aaa42d3a652473db5956a92437c08e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aaa42d3a652473db5956a92437c08e3" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42505 } }
I20260131 03:12:17.422714 1194 leader_election.cc:304] T 00000000000000000000000000000000 P 4aaa42d3a652473db5956a92437c08e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4aaa42d3a652473db5956a92437c08e3; no voters:
I20260131 03:12:17.422771 19792 external_mini_cluster.cc:1468] Setting key ac7b93fb41b3068fb9b7503e0f6a6423
I20260131 03:12:17.422876 1194 leader_election.cc:290] T 00000000000000000000000000000000 P 4aaa42d3a652473db5956a92437c08e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:12:17.423069 1198 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4aaa42d3a652473db5956a92437c08e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:12:17.423111 1194 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4aaa42d3a652473db5956a92437c08e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:12:17.423249 1198 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4aaa42d3a652473db5956a92437c08e3 [term 1 LEADER]: Becoming Leader. State: Replica: 4aaa42d3a652473db5956a92437c08e3, State: Running, Role: LEADER
I20260131 03:12:17.423336 1198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4aaa42d3a652473db5956a92437c08e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aaa42d3a652473db5956a92437c08e3" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42505 } }
I20260131 03:12:17.423717 1197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aaa42d3a652473db5956a92437c08e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4aaa42d3a652473db5956a92437c08e3. Latest consensus state: current_term: 1 leader_uuid: "4aaa42d3a652473db5956a92437c08e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aaa42d3a652473db5956a92437c08e3" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42505 } } }
I20260131 03:12:17.423802 1197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aaa42d3a652473db5956a92437c08e3 [sys.catalog]: This master's current role is: LEADER
I20260131 03:12:17.423877 1199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aaa42d3a652473db5956a92437c08e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4aaa42d3a652473db5956a92437c08e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aaa42d3a652473db5956a92437c08e3" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42505 } } }
I20260131 03:12:17.423952 1199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aaa42d3a652473db5956a92437c08e3 [sys.catalog]: This master's current role is: LEADER
I20260131 03:12:17.424299 1208 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:12:17.424779 1208 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:12:17.427790 1208 catalog_manager.cc:1357] Generated new cluster ID: dfbbe52e33b54f6fa03b9aa4c77c9a8b
I20260131 03:12:17.427857 1208 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Jan 31 03:12:17 dist-test-slave-5nnp krb5kdc[1110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:12:17.433214 1201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:17.423234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45118 (local address 127.19.84.62:42505)
0131 03:12:17.423384 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:17.423424 (+ 40us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:17.423767 (+ 343us) server_negotiation.cc:408] Connection header received
0131 03:12:17.423952 (+ 185us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:17.423958 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:17.424011 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:17.424150 (+ 139us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:17.425292 (+ 1142us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:17.426285 (+ 993us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:17.427073 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:17.427280 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:17.429789 (+ 2509us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:17.429794 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:17.429795 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:17.429816 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:17.431815 (+ 1999us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:17.432175 (+ 360us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:17.432178 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:17.432178 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:17.432251 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:17.432418 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:17.432419 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:17.432419 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:17.432668 (+ 249us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:17.432817 (+ 149us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:17.432899 (+ 82us) server_negotiation.cc:300] Negotiation successful
0131 03:12:17.433025 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260131 03:12:17.434162 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:12:17.444480 1208 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:12:17.445116 1208 catalog_manager.cc:1514] Loading token signing keys...
Jan 31 03:12:17 dist-test-slave-5nnp krb5kdc[1110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260131 03:12:17.451934 1208 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 4aaa42d3a652473db5956a92437c08e3: Generated new TSK 0
I20260131 03:12:17.452102 1208 catalog_manager.cc:1524] Initializing in-progress tserver states...
Jan 31 03:12:17 dist-test-slave-5nnp krb5kdc[1110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.19.84.62@KRBTEST.COM
I20260131 03:12:17.461820 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 1126
2026-01-31T03:12:17Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2300 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:12:17 dist-test-slave-5nnp krb5kdc[1223](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:12:17 dist-test-slave-5nnp krb5kdc[1223](info): set up 2 sockets
Jan 31 03:12:17 dist-test-slave-5nnp krb5kdc[1223](info): commencing operation
krb5kdc: starting...
W20260131 03:12:19.463894 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.984s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:12:19 dist-test-slave-5nnp krb5kdc[1223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:12:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:12:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:12:19.598227 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:41917
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:41917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:12:19.676429 1239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:19.676651 1239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:19.676687 1239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:19.678129 1239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:12:19.678211 1239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:19.678253 1239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:19.678285 1239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:12:19.678310 1239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:12:19.680047 1239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:41917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:41917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:19.680285 1239 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:19.680486 1239 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:19.682716 1244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:19.682723 1245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:19.682725 1247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:19.683281 1239 server_base.cc:1060] running on GCE node
I20260131 03:12:19.683537 1239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:19.683732 1239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:19.684859 1239 hybrid_clock.cc:648] HybridClock initialized: now 1769829139684846 us; error 27 us; skew 500 ppm
Jan 31 03:12:19 dist-test-slave-5nnp krb5kdc[1223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829139, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:19.687024 1239 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:12:19.687587 1239 webserver.cc:492] Webserver started at http://127.19.84.62:32973/ using document root <none> and password file <none>
I20260131 03:12:19.687741 1239 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:19.687772 1239 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:19.687846 1239 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:12:19.688766 1239 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "ec75413d680c4c8c9f4b89582b18d590"
format_stamp: "Formatted at 2026-01-31 03:12:19 on dist-test-slave-5nnp"
server_key: "a5189e440d723a37e47f1e67bd8ac5fe"
server_key_iv: "5121024338d3884ff6c1d3fd147418b7"
server_key_version: "encryptionkey@0"
I20260131 03:12:19.689025 1239 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "ec75413d680c4c8c9f4b89582b18d590"
format_stamp: "Formatted at 2026-01-31 03:12:19 on dist-test-slave-5nnp"
server_key: "a5189e440d723a37e47f1e67bd8ac5fe"
server_key_iv: "5121024338d3884ff6c1d3fd147418b7"
server_key_version: "encryptionkey@0"
I20260131 03:12:19.690135 1239 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:12:19.690764 1254 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:19.691058 1239 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:12:19.691154 1239 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "ec75413d680c4c8c9f4b89582b18d590"
format_stamp: "Formatted at 2026-01-31 03:12:19 on dist-test-slave-5nnp"
server_key: "a5189e440d723a37e47f1e67bd8ac5fe"
server_key_iv: "5121024338d3884ff6c1d3fd147418b7"
server_key_version: "encryptionkey@0"
I20260131 03:12:19.691223 1239 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:19.711496 1239 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:19.716953 1239 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:19.717123 1239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:19.721272 1239 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:41917
I20260131 03:12:19.721311 1306 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:41917 every 8 connection(s)
I20260131 03:12:19.721666 1239 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:12:19.722276 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 1239
I20260131 03:12:19.722406 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:12:19.722431 1307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:12:19.722594 19792 external_mini_cluster.cc:1468] Setting key 8f32b46e2758101dce55344d97a0efd4
I20260131 03:12:19.726009 1307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590: Bootstrap starting.
I20260131 03:12:19.726694 1307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590: Neither blocks nor log segments found. Creating new log.
I20260131 03:12:19.726958 1307 log.cc:826] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:19.727804 1307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590: No bootstrap required, opened a new log
I20260131 03:12:19.728838 1307 raft_consensus.cc:359] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec75413d680c4c8c9f4b89582b18d590" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41917 } }
I20260131 03:12:19.728953 1307 raft_consensus.cc:385] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:12:19.728974 1307 raft_consensus.cc:740] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec75413d680c4c8c9f4b89582b18d590, State: Initialized, Role: FOLLOWER
Jan 31 03:12:19 dist-test-slave-5nnp krb5kdc[1223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:12:19.729115 1307 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec75413d680c4c8c9f4b89582b18d590" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41917 } }
I20260131 03:12:19.729212 1307 raft_consensus.cc:399] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:12:19.729259 1307 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:12:19.729312 1307 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:12:19.730038 1307 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec75413d680c4c8c9f4b89582b18d590" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41917 } }
I20260131 03:12:19.730158 1307 leader_election.cc:304] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec75413d680c4c8c9f4b89582b18d590; no voters:
I20260131 03:12:19.730329 1307 leader_election.cc:290] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:12:19.730413 1312 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:12:19.730572 1307 sys_catalog.cc:565] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:12:19.730607 1312 raft_consensus.cc:697] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 1 LEADER]: Becoming Leader. State: Replica: ec75413d680c4c8c9f4b89582b18d590, State: Running, Role: LEADER
I20260131 03:12:19.730823 1312 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec75413d680c4c8c9f4b89582b18d590" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41917 } }
I20260131 03:12:19.731875 1314 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec75413d680c4c8c9f4b89582b18d590. Latest consensus state: current_term: 1 leader_uuid: "ec75413d680c4c8c9f4b89582b18d590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec75413d680c4c8c9f4b89582b18d590" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41917 } } }
I20260131 03:12:19.731986 1314 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [sys.catalog]: This master's current role is: LEADER
I20260131 03:12:19.732368 1313 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec75413d680c4c8c9f4b89582b18d590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec75413d680c4c8c9f4b89582b18d590" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41917 } } }
I20260131 03:12:19.732448 1313 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [sys.catalog]: This master's current role is: LEADER
W20260131 03:12:19.732683 1328 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:12:19.732740 1328 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:12:19.732784 1327 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:12:19.732939 1327 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:12:19.733979 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:19.723113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58926 (local address 127.19.84.62:41917)
0131 03:12:19.723301 (+ 188us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:19.723310 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:19.723329 (+ 19us) server_negotiation.cc:408] Connection header received
0131 03:12:19.724395 (+ 1066us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:19.724403 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:19.724460 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:19.724564 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:19.725553 (+ 989us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:19.726337 (+ 784us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:19.727118 (+ 781us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:19.727411 (+ 293us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:19.729590 (+ 2179us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:19.729597 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:19.729597 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:19.729618 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:19.732506 (+ 2888us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:19.732882 (+ 376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:19.732884 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:19.732885 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:19.732947 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:19.733126 (+ 179us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:19.733127 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:19.733127 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:19.733321 (+ 194us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:19.733529 (+ 208us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:19.733607 (+ 78us) server_negotiation.cc:300] Negotiation successful
0131 03:12:19.733783 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":107,"threads_started":1}
I20260131 03:12:19.734228 1327 catalog_manager.cc:1357] Generated new cluster ID: 4240b6fddefc4a979e867c316c62a1ee
I20260131 03:12:19.734280 1327 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:12:19.747452 1327 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:12:19.748247 1327 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:12:19.762145 1327 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590: Generated new TSK 0
I20260131 03:12:19.762374 1327 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:12:19.814312 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:41917
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:12:19.892808 1335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:19.893005 1335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:19.893043 1335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:19.894495 1335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:19.894564 1335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:19.894606 1335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:12:19.896303 1335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:41917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:19.896544 1335 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:19.896736 1335 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:19.899355 1340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:19.899430 1341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:19.899355 1343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:19.899688 1335 server_base.cc:1060] running on GCE node
I20260131 03:12:19.899874 1335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:19.900079 1335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:19.901243 1335 hybrid_clock.cc:648] HybridClock initialized: now 1769829139901216 us; error 37 us; skew 500 ppm
Jan 31 03:12:19 dist-test-slave-5nnp krb5kdc[1223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829139, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:19.903493 1335 init.cc:377] Logged in from keytab as kudu/127.19.84.1@KRBTEST.COM (short username kudu)
I20260131 03:12:19.904026 1335 webserver.cc:492] Webserver started at http://127.19.84.1:36141/ using document root <none> and password file <none>
I20260131 03:12:19.904174 1335 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:19.904211 1335 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:19.904289 1335 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:12:19.905148 1335 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "90c0155151a74528816bcf8a9287396d"
format_stamp: "Formatted at 2026-01-31 03:12:19 on dist-test-slave-5nnp"
server_key: "2e9c122da4dcd1205743b00fa4290596"
server_key_iv: "cfa83252d1cd7a1bd836a91672e9f84a"
server_key_version: "encryptionkey@0"
I20260131 03:12:19.905414 1335 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "90c0155151a74528816bcf8a9287396d"
format_stamp: "Formatted at 2026-01-31 03:12:19 on dist-test-slave-5nnp"
server_key: "2e9c122da4dcd1205743b00fa4290596"
server_key_iv: "cfa83252d1cd7a1bd836a91672e9f84a"
server_key_version: "encryptionkey@0"
I20260131 03:12:19.906550 1335 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:12:19.907178 1350 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:19.907444 1335 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:12:19.907507 1335 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "90c0155151a74528816bcf8a9287396d"
format_stamp: "Formatted at 2026-01-31 03:12:19 on dist-test-slave-5nnp"
server_key: "2e9c122da4dcd1205743b00fa4290596"
server_key_iv: "cfa83252d1cd7a1bd836a91672e9f84a"
server_key_version: "encryptionkey@0"
I20260131 03:12:19.907557 1335 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:19.930619 1335 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:19.932504 1335 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:19.932612 1335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:19.932792 1335 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:12:19.933125 1335 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:12:19.933158 1335 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:19.933182 1335 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:12:19.933238 1335 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:19.938966 1335 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:35801
I20260131 03:12:19.939100 1463 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:35801 every 8 connection(s)
I20260131 03:12:19.939353 1335 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
Jan 31 03:12:19 dist-test-slave-5nnp krb5kdc[1223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829139, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:12:19.947614 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:19.940087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:52683 (local address 127.19.84.62:41917)
0131 03:12:19.940154 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:19.940156 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:19.940342 (+ 186us) server_negotiation.cc:408] Connection header received
0131 03:12:19.940806 (+ 464us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:19.940808 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:19.940817 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:19.940877 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:19.941627 (+ 750us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:19.942124 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:19.942654 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:19.942767 (+ 113us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:19.945124 (+ 2357us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:19.945131 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:19.945131 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:19.945138 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:19.946528 (+ 1390us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:19.946846 (+ 318us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:19.946847 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:19.946847 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:19.946866 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:19.947033 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:19.947034 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:19.947034 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:19.947118 (+ 84us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:19.947168 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:19.947463 (+ 295us) server_negotiation.cc:300] Negotiation successful
0131 03:12:19.947545 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260131 03:12:19.947681 1466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:19.940041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.1:52683)
0131 03:12:19.940226 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:19.940259 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:19.940731 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:19.940940 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:19.940944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:19.941267 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:19.941549 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:19.941552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:19.942211 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:19.942211 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:19.942583 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:19.942585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:19.942687 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:19.943618 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:19.943635 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:19.945033 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:19.946601 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:19.946605 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:19.946607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:19.946784 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:19.946918 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:19.946918 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:19.946919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:19.946984 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:19.947169 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:19.947173 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:19.947280 (+ 107us) client_negotiation.cc:769] Sending connection context
0131 03:12:19.947416 (+ 136us) client_negotiation.cc:240] Negotiation successful
0131 03:12:19.947544 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260131 03:12:19.948235 1464 heartbeater.cc:344] Connected to a master server at 127.19.84.62:41917
I20260131 03:12:19.948319 1464 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:19.948479 1464 heartbeater.cc:507] Master 127.19.84.62:41917 requested a full tablet report, sending...
I20260131 03:12:19.948889 1271 ts_manager.cc:194] Registered new tserver with Master: 90c0155151a74528816bcf8a9287396d (127.19.84.1:35801)
I20260131 03:12:19.949222 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 1335
I20260131 03:12:19.949327 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:12:19.949515 19792 external_mini_cluster.cc:1468] Setting key 04b638078ef6fb0a7d699a258e032fbc
I20260131 03:12:19.949690 1271 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.1@KRBTEST.COM'} at 127.19.84.1:52683
WARNING: no policy specified for kudu/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:12:19.998036 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:41917
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:12:20.084643 1471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:20.084839 1471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:20.084864 1471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:20.086447 1471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:20.086501 1471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:20.086529 1471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:12:20.088286 1471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:41917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:20.088536 1471 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:20.088737 1471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:20.091332 1479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:20.091307 1477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:20.091507 1471 server_base.cc:1060] running on GCE node
W20260131 03:12:20.091325 1476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:20.091713 1471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:20.091941 1471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:20.093096 1471 hybrid_clock.cc:648] HybridClock initialized: now 1769829140093081 us; error 25 us; skew 500 ppm
Jan 31 03:12:20 dist-test-slave-5nnp krb5kdc[1223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829140, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:20.095470 1471 init.cc:377] Logged in from keytab as kudu/127.19.84.2@KRBTEST.COM (short username kudu)
I20260131 03:12:20.095991 1471 webserver.cc:492] Webserver started at http://127.19.84.2:34399/ using document root <none> and password file <none>
I20260131 03:12:20.096189 1471 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:20.096246 1471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:20.096369 1471 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:12:20.097334 1471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "f9f2dc8e777c45fc971651de8f01f690"
format_stamp: "Formatted at 2026-01-31 03:12:20 on dist-test-slave-5nnp"
server_key: "c260a49d134b2599aee18aacbcca8528"
server_key_iv: "95cfd6c7349fb3d6780b261f08408e1a"
server_key_version: "encryptionkey@0"
I20260131 03:12:20.097661 1471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "f9f2dc8e777c45fc971651de8f01f690"
format_stamp: "Formatted at 2026-01-31 03:12:20 on dist-test-slave-5nnp"
server_key: "c260a49d134b2599aee18aacbcca8528"
server_key_iv: "95cfd6c7349fb3d6780b261f08408e1a"
server_key_version: "encryptionkey@0"
I20260131 03:12:20.099042 1471 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:12:20.099944 1486 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:20.100207 1471 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260131 03:12:20.100303 1471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "f9f2dc8e777c45fc971651de8f01f690"
format_stamp: "Formatted at 2026-01-31 03:12:20 on dist-test-slave-5nnp"
server_key: "c260a49d134b2599aee18aacbcca8528"
server_key_iv: "95cfd6c7349fb3d6780b261f08408e1a"
server_key_version: "encryptionkey@0"
I20260131 03:12:20.100404 1471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:20.110706 1471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:20.113047 1471 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:20.113237 1471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:20.113492 1471 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:12:20.113827 1471 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:12:20.113883 1471 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:20.113930 1471 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:12:20.113960 1471 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:20.119925 1471 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:42987
I20260131 03:12:20.119998 1599 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:42987 every 8 connection(s)
I20260131 03:12:20.120330 1471 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
I20260131 03:12:20.124200 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 1471
I20260131 03:12:20.124325 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:12:20.124534 19792 external_mini_cluster.cc:1468] Setting key e84a8eb739610fb384cba08696e0af02
Jan 31 03:12:20 dist-test-slave-5nnp krb5kdc[1223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829140, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:12:20.130317 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.121277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:53959 (local address 127.19.84.62:41917)
0131 03:12:20.121353 (+ 76us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:20.121356 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:20.121530 (+ 174us) server_negotiation.cc:408] Connection header received
0131 03:12:20.122066 (+ 536us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:20.122067 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:20.122082 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:20.122127 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:20.122958 (+ 831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.123600 (+ 642us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.124311 (+ 711us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.124486 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.127260 (+ 2774us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:20.127264 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:20.127265 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:20.127272 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:20.128974 (+ 1702us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:20.129388 (+ 414us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:20.129390 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:20.129390 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:20.129424 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:20.129671 (+ 247us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:20.129672 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:20.129672 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:20.129775 (+ 103us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:20.129851 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:20.130124 (+ 273us) server_negotiation.cc:300] Negotiation successful
0131 03:12:20.130226 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260131 03:12:20.130455 1602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.121181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.2:53959)
0131 03:12:20.121398 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:20.121448 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:20.121954 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:20.122228 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:20.122232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:20.122565 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:20.122853 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.122856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.123690 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.123692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:20.124156 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.124159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.124327 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.125532 (+ 1205us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:20.125549 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:20.127154 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:20.129061 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:20.129067 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:20.129069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:20.129301 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:20.129516 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:20.129517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:20.129517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:20.129599 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:20.129839 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:20.129844 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:20.129977 (+ 133us) client_negotiation.cc:769] Sending connection context
0131 03:12:20.130110 (+ 133us) client_negotiation.cc:240] Negotiation successful
0131 03:12:20.130257 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":107,"threads_started":1}
I20260131 03:12:20.130882 1600 heartbeater.cc:344] Connected to a master server at 127.19.84.62:41917
I20260131 03:12:20.130970 1600 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:20.131182 1600 heartbeater.cc:507] Master 127.19.84.62:41917 requested a full tablet report, sending...
I20260131 03:12:20.131593 1270 ts_manager.cc:194] Registered new tserver with Master: f9f2dc8e777c45fc971651de8f01f690 (127.19.84.2:42987)
I20260131 03:12:20.132005 1270 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.2@KRBTEST.COM'} at 127.19.84.2:53959
WARNING: no policy specified for kudu/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:12:20.173856 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:41917
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:12:20.252527 1607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:20.252712 1607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:20.252737 1607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:20.254220 1607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:20.254267 1607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:20.254295 1607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:12:20.255908 1607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:41917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:20.256139 1607 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:20.256342 1607 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:20.259020 1615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:20.259020 1613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:20.259069 1612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:20.259326 1607 server_base.cc:1060] running on GCE node
I20260131 03:12:20.259524 1607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:20.259765 1607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:20.260915 1607 hybrid_clock.cc:648] HybridClock initialized: now 1769829140260890 us; error 47 us; skew 500 ppm
Jan 31 03:12:20 dist-test-slave-5nnp krb5kdc[1223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829140, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:20.263438 1607 init.cc:377] Logged in from keytab as kudu/127.19.84.3@KRBTEST.COM (short username kudu)
I20260131 03:12:20.263976 1607 webserver.cc:492] Webserver started at http://127.19.84.3:38147/ using document root <none> and password file <none>
I20260131 03:12:20.264184 1607 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:20.264250 1607 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:20.264382 1607 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:12:20.265321 1607 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "2c2bad11ca33403eb47410763aa631e6"
format_stamp: "Formatted at 2026-01-31 03:12:20 on dist-test-slave-5nnp"
server_key: "355a3d381a466f666c5140d09ec65643"
server_key_iv: "2cca9d643fb5fdb2cd070cc33caff6e2"
server_key_version: "encryptionkey@0"
I20260131 03:12:20.265650 1607 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "2c2bad11ca33403eb47410763aa631e6"
format_stamp: "Formatted at 2026-01-31 03:12:20 on dist-test-slave-5nnp"
server_key: "355a3d381a466f666c5140d09ec65643"
server_key_iv: "2cca9d643fb5fdb2cd070cc33caff6e2"
server_key_version: "encryptionkey@0"
I20260131 03:12:20.266839 1607 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:12:20.267764 1622 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:20.267987 1607 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:12:20.268067 1607 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "2c2bad11ca33403eb47410763aa631e6"
format_stamp: "Formatted at 2026-01-31 03:12:20 on dist-test-slave-5nnp"
server_key: "355a3d381a466f666c5140d09ec65643"
server_key_iv: "2cca9d643fb5fdb2cd070cc33caff6e2"
server_key_version: "encryptionkey@0"
I20260131 03:12:20.268146 1607 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:20.274991 1607 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:20.277010 1607 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:20.277168 1607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:20.277390 1607 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:12:20.277710 1607 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:12:20.277765 1607 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:20.277810 1607 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:12:20.277833 1607 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:20.283864 1607 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:39851
I20260131 03:12:20.283936 1735 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:39851 every 8 connection(s)
I20260131 03:12:20.284240 1607 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
I20260131 03:12:20.288189 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 1607
I20260131 03:12:20.288321 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:12:20.288520 19792 external_mini_cluster.cc:1468] Setting key 1f701712306c454c467b6afab4ec7c69
Jan 31 03:12:20 dist-test-slave-5nnp krb5kdc[1223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829140, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:12:20.293134 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.285051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:37667 (local address 127.19.84.62:41917)
0131 03:12:20.285125 (+ 74us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:20.285128 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:20.285308 (+ 180us) server_negotiation.cc:408] Connection header received
0131 03:12:20.285755 (+ 447us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:20.285756 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:20.285765 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:20.285817 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:20.286585 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.287064 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.287690 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.287807 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.290417 (+ 2610us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:20.290420 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:20.290421 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:20.290427 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:20.291790 (+ 1363us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:20.292125 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:20.292126 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:20.292127 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:20.292150 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:20.292394 (+ 244us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:20.292394 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:20.292395 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:20.292486 (+ 91us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:20.292555 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:20.292978 (+ 423us) server_negotiation.cc:300] Negotiation successful
0131 03:12:20.293054 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260131 03:12:20.293247 1738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.285004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.3:37667)
0131 03:12:20.285168 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:20.285219 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:20.285678 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:20.285871 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:20.285874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:20.286174 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:20.286486 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.286489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.287141 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.287141 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:20.287611 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.287612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.287723 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.288742 (+ 1019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:20.288758 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:20.290342 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:20.291858 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:20.291863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:20.291866 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:20.292053 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:20.292252 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:20.292253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:20.292254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:20.292334 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:20.292680 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:20.292686 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:20.292848 (+ 162us) client_negotiation.cc:769] Sending connection context
0131 03:12:20.292953 (+ 105us) client_negotiation.cc:240] Negotiation successful
0131 03:12:20.293058 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260131 03:12:20.293708 1736 heartbeater.cc:344] Connected to a master server at 127.19.84.62:41917
I20260131 03:12:20.293814 1736 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:20.294032 1736 heartbeater.cc:507] Master 127.19.84.62:41917 requested a full tablet report, sending...
I20260131 03:12:20.294438 1270 ts_manager.cc:194] Registered new tserver with Master: 2c2bad11ca33403eb47410763aa631e6 (127.19.84.3:39851)
I20260131 03:12:20.294828 1270 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:37667
I20260131 03:12:20.300491 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:12:20.306993 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.301795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58934 (local address 127.19.84.62:41917)
0131 03:12:20.301855 (+ 60us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:20.301857 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:20.301919 (+ 62us) server_negotiation.cc:408] Connection header received
0131 03:12:20.301974 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:20.301975 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:20.301982 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:20.302021 (+ 39us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:20.302663 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.303112 (+ 449us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.303753 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.303868 (+ 115us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.304855 (+ 987us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:20.304858 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:20.304858 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:20.304863 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:20.306014 (+ 1151us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:20.306387 (+ 373us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:20.306388 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:20.306388 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:20.306407 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:20.306536 (+ 129us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:20.306537 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:20.306537 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:20.306642 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:20.306718 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:20.306864 (+ 146us) server_negotiation.cc:300] Negotiation successful
0131 03:12:20.306928 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260131 03:12:20.308669 1270 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:12:20.309865 1270 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:12:20.315279 1751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.312198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:42987 (local address 127.0.0.1:39684)
0131 03:12:20.312363 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:20.312373 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:20.312432 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:20.312757 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:20.312758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:20.312759 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:20.312944 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.312946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.314238 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.314238 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:20.315050 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.315052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.315113 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.315116 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:12:20.315147 (+ 31us) client_negotiation.cc:240] Negotiation successful
0131 03:12:20.315199 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":62,"threads_started":1}
I20260131 03:12:20.315435 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.311994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:39851 (local address 127.0.0.1:40936)
0131 03:12:20.312112 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:20.312144 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:20.312224 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:20.312635 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:20.312638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:20.312640 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:20.312913 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.312915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.313991 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.313992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:20.314858 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.314860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.314933 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.314969 (+ 36us) client_negotiation.cc:769] Sending connection context
0131 03:12:20.315014 (+ 45us) client_negotiation.cc:240] Negotiation successful
0131 03:12:20.315053 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":70,"threads_started":1}
I20260131 03:12:20.315855 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.312167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40936 (local address 127.19.84.3:39851)
0131 03:12:20.312483 (+ 316us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:20.312491 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:20.312501 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:12:20.312538 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:20.312540 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:20.312557 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:20.312625 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:20.312999 (+ 374us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.313902 (+ 903us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.314962 (+ 1060us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.315655 (+ 693us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.315708 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:20.315741 (+ 33us) server_negotiation.cc:300] Negotiation successful
0131 03:12:20.315781 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":92,"threads_started":1}
I20260131 03:12:20.316222 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.312025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:35801 (local address 127.0.0.1:34726)
0131 03:12:20.312322 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:20.312331 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:20.312723 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:20.312987 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:20.312987 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:20.312989 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:20.313356 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.313358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.314542 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.314543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:20.315295 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.315298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.316040 (+ 742us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.316044 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:12:20.316121 (+ 77us) client_negotiation.cc:240] Negotiation successful
0131 03:12:20.316163 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":77,"threads_started":1}
I20260131 03:12:20.316463 1670 tablet_service.cc:1505] Processing CreateTablet for tablet b2306eae776c4627abcb72119604a470 (DEFAULT_TABLE table=test-table [id=6954671bacc449418b609c9821288dfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:12:20.316572 1752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.312415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.19.84.2:42987)
0131 03:12:20.312602 (+ 187us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:20.312608 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:20.312616 (+ 8us) server_negotiation.cc:408] Connection header received
0131 03:12:20.312647 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:20.312648 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:20.312661 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:20.312737 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:20.313021 (+ 284us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.314154 (+ 1133us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.315834 (+ 1680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.316386 (+ 552us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.316430 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:20.316469 (+ 39us) server_negotiation.cc:300] Negotiation successful
0131 03:12:20.316515 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":114,"threads_started":1}
I20260131 03:12:20.316771 1670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2306eae776c4627abcb72119604a470. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:12:20.317087 1534 tablet_service.cc:1505] Processing CreateTablet for tablet b2306eae776c4627abcb72119604a470 (DEFAULT_TABLE table=test-table [id=6954671bacc449418b609c9821288dfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:12:20.317385 1534 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2306eae776c4627abcb72119604a470. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:12:20.319154 1753 tablet_bootstrap.cc:492] T b2306eae776c4627abcb72119604a470 P 2c2bad11ca33403eb47410763aa631e6: Bootstrap starting.
I20260131 03:12:20.319339 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.312128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.19.84.1:35801)
0131 03:12:20.312307 (+ 179us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:20.312316 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:20.312393 (+ 77us) server_negotiation.cc:408] Connection header received
0131 03:12:20.312825 (+ 432us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:20.312828 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:20.312843 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:20.312881 (+ 38us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:20.313424 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.314452 (+ 1028us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.315444 (+ 992us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.315980 (+ 536us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.316014 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:20.319186 (+ 3172us) server_negotiation.cc:300] Negotiation successful
0131 03:12:20.319261 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260131 03:12:20.319825 1753 tablet_bootstrap.cc:654] T b2306eae776c4627abcb72119604a470 P 2c2bad11ca33403eb47410763aa631e6: Neither blocks nor log segments found. Creating new log.
I20260131 03:12:20.319950 1398 tablet_service.cc:1505] Processing CreateTablet for tablet b2306eae776c4627abcb72119604a470 (DEFAULT_TABLE table=test-table [id=6954671bacc449418b609c9821288dfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:12:20.320154 1753 log.cc:826] T b2306eae776c4627abcb72119604a470 P 2c2bad11ca33403eb47410763aa631e6: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:20.320253 1398 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2306eae776c4627abcb72119604a470. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:12:20.320964 1753 tablet_bootstrap.cc:492] T b2306eae776c4627abcb72119604a470 P 2c2bad11ca33403eb47410763aa631e6: No bootstrap required, opened a new log
I20260131 03:12:20.321034 1753 ts_tablet_manager.cc:1403] T b2306eae776c4627abcb72119604a470 P 2c2bad11ca33403eb47410763aa631e6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260131 03:12:20.322651 1753 raft_consensus.cc:359] T b2306eae776c4627abcb72119604a470 P 2c2bad11ca33403eb47410763aa631e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } } peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } }
I20260131 03:12:20.322808 1753 raft_consensus.cc:385] T b2306eae776c4627abcb72119604a470 P 2c2bad11ca33403eb47410763aa631e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:12:20.322840 1753 raft_consensus.cc:740] T b2306eae776c4627abcb72119604a470 P 2c2bad11ca33403eb47410763aa631e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c2bad11ca33403eb47410763aa631e6, State: Initialized, Role: FOLLOWER
I20260131 03:12:20.322934 1753 consensus_queue.cc:260] T b2306eae776c4627abcb72119604a470 P 2c2bad11ca33403eb47410763aa631e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } } peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } }
I20260131 03:12:20.323163 1753 ts_tablet_manager.cc:1434] T b2306eae776c4627abcb72119604a470 P 2c2bad11ca33403eb47410763aa631e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260131 03:12:20.323273 1754 tablet_bootstrap.cc:492] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690: Bootstrap starting.
I20260131 03:12:20.323285 1736 heartbeater.cc:499] Master 127.19.84.62:41917 was elected leader, sending a full tablet report...
I20260131 03:12:20.323334 1756 tablet_bootstrap.cc:492] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d: Bootstrap starting.
I20260131 03:12:20.323963 1754 tablet_bootstrap.cc:654] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690: Neither blocks nor log segments found. Creating new log.
I20260131 03:12:20.323963 1756 tablet_bootstrap.cc:654] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d: Neither blocks nor log segments found. Creating new log.
I20260131 03:12:20.324276 1756 log.cc:826] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:20.324290 1754 log.cc:826] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:20.325071 1754 tablet_bootstrap.cc:492] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690: No bootstrap required, opened a new log
I20260131 03:12:20.325145 1754 ts_tablet_manager.cc:1403] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260131 03:12:20.325183 1756 tablet_bootstrap.cc:492] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d: No bootstrap required, opened a new log
I20260131 03:12:20.325235 1756 ts_tablet_manager.cc:1403] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:12:20.326767 1756 raft_consensus.cc:359] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } } peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } }
I20260131 03:12:20.326749 1754 raft_consensus.cc:359] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } } peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } }
I20260131 03:12:20.326907 1754 raft_consensus.cc:385] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:12:20.326907 1756 raft_consensus.cc:385] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:12:20.326938 1754 raft_consensus.cc:740] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9f2dc8e777c45fc971651de8f01f690, State: Initialized, Role: FOLLOWER
I20260131 03:12:20.326938 1756 raft_consensus.cc:740] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90c0155151a74528816bcf8a9287396d, State: Initialized, Role: FOLLOWER
I20260131 03:12:20.327059 1754 consensus_queue.cc:260] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } } peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } }
I20260131 03:12:20.327054 1756 consensus_queue.cc:260] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } } peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } }
I20260131 03:12:20.327315 1756 ts_tablet_manager.cc:1434] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:12:20.327315 1754 ts_tablet_manager.cc:1434] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:12:20.327455 1464 heartbeater.cc:499] Master 127.19.84.62:41917 was elected leader, sending a full tablet report...
I20260131 03:12:20.327726 1600 heartbeater.cc:499] Master 127.19.84.62:41917 was elected leader, sending a full tablet report...
I20260131 03:12:20.332381 1760 raft_consensus.cc:493] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:12:20.332472 1760 raft_consensus.cc:515] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } } peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } }
I20260131 03:12:20.332784 1760 leader_election.cc:290] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c2bad11ca33403eb47410763aa631e6 (127.19.84.3:39851), 90c0155151a74528816bcf8a9287396d (127.19.84.1:35801)
I20260131 03:12:20.336210 1602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.332917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:39851 (local address 127.19.84.2:52395)
0131 03:12:20.332978 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:20.332991 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:20.333037 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:20.333316 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:20.333317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:20.333320 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:20.333578 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.333581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.335004 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.335005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:20.335966 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.335969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.336052 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.336057 (+ 5us) client_negotiation.cc:769] Sending connection context
0131 03:12:20.336094 (+ 37us) client_negotiation.cc:240] Negotiation successful
0131 03:12:20.336128 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260131 03:12:20.336407 1762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.332980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:35801 (local address 127.19.84.2:46943)
0131 03:12:20.333199 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:20.333210 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:20.333269 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:20.333416 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:20.333417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:20.333418 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:20.333722 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.333724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.335004 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.335005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:20.335966 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.335969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.336054 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.336057 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:12:20.336094 (+ 37us) client_negotiation.cc:240] Negotiation successful
0131 03:12:20.336128 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":98,"threads_started":1}
I20260131 03:12:20.336670 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.332981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:52395 (local address 127.19.84.3:39851)
0131 03:12:20.333027 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:20.333029 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:20.333041 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:12:20.333147 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:20.333149 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:20.333162 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:20.333233 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:20.333662 (+ 429us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.334892 (+ 1230us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.336063 (+ 1171us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.336559 (+ 496us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.336567 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:20.336587 (+ 20us) server_negotiation.cc:300] Negotiation successful
0131 03:12:20.336614 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260131 03:12:20.336668 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.333053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:46943 (local address 127.19.84.1:35801)
0131 03:12:20.333094 (+ 41us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:20.333096 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:20.333295 (+ 199us) server_negotiation.cc:408] Connection header received
0131 03:12:20.333321 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:20.333322 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:20.333336 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:20.333408 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:20.333789 (+ 381us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.334891 (+ 1102us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.336063 (+ 1172us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.336540 (+ 477us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.336549 (+ 9us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:20.336574 (+ 25us) server_negotiation.cc:300] Negotiation successful
0131 03:12:20.336606 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260131 03:12:20.336933 1418 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2306eae776c4627abcb72119604a470" candidate_uuid: "f9f2dc8e777c45fc971651de8f01f690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c0155151a74528816bcf8a9287396d" is_pre_election: true
I20260131 03:12:20.336970 1690 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2306eae776c4627abcb72119604a470" candidate_uuid: "f9f2dc8e777c45fc971651de8f01f690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c2bad11ca33403eb47410763aa631e6" is_pre_election: true
I20260131 03:12:20.337071 1418 raft_consensus.cc:2468] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9f2dc8e777c45fc971651de8f01f690 in term 0.
I20260131 03:12:20.337121 1690 raft_consensus.cc:2468] T b2306eae776c4627abcb72119604a470 P 2c2bad11ca33403eb47410763aa631e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9f2dc8e777c45fc971651de8f01f690 in term 0.
I20260131 03:12:20.337318 1490 leader_election.cc:304] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90c0155151a74528816bcf8a9287396d, f9f2dc8e777c45fc971651de8f01f690; no voters:
I20260131 03:12:20.337504 1760 raft_consensus.cc:2804] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:12:20.337568 1760 raft_consensus.cc:493] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:12:20.337601 1760 raft_consensus.cc:3060] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:12:20.338213 1760 raft_consensus.cc:515] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } } peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } }
I20260131 03:12:20.338411 1760 leader_election.cc:290] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690 [CANDIDATE]: Term 1 election: Requested vote from peers 2c2bad11ca33403eb47410763aa631e6 (127.19.84.3:39851), 90c0155151a74528816bcf8a9287396d (127.19.84.1:35801)
I20260131 03:12:20.338507 1418 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2306eae776c4627abcb72119604a470" candidate_uuid: "f9f2dc8e777c45fc971651de8f01f690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c0155151a74528816bcf8a9287396d"
I20260131 03:12:20.338590 1418 raft_consensus.cc:3060] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:12:20.338572 1690 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2306eae776c4627abcb72119604a470" candidate_uuid: "f9f2dc8e777c45fc971651de8f01f690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c2bad11ca33403eb47410763aa631e6"
I20260131 03:12:20.338665 1690 raft_consensus.cc:3060] T b2306eae776c4627abcb72119604a470 P 2c2bad11ca33403eb47410763aa631e6 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:12:20.339315 1418 raft_consensus.cc:2468] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9f2dc8e777c45fc971651de8f01f690 in term 1.
I20260131 03:12:20.339327 1690 raft_consensus.cc:2468] T b2306eae776c4627abcb72119604a470 P 2c2bad11ca33403eb47410763aa631e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9f2dc8e777c45fc971651de8f01f690 in term 1.
I20260131 03:12:20.339547 1487 leader_election.cc:304] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c2bad11ca33403eb47410763aa631e6, f9f2dc8e777c45fc971651de8f01f690; no voters:
I20260131 03:12:20.339701 1760 raft_consensus.cc:2804] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:12:20.339924 1760 raft_consensus.cc:697] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690 [term 1 LEADER]: Becoming Leader. State: Replica: f9f2dc8e777c45fc971651de8f01f690, State: Running, Role: LEADER
I20260131 03:12:20.340044 1760 consensus_queue.cc:237] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } } peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } }
I20260131 03:12:20.340975 1271 catalog_manager.cc:5654] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9f2dc8e777c45fc971651de8f01f690 (127.19.84.2). New cstate: current_term: 1 leader_uuid: "f9f2dc8e777c45fc971651de8f01f690" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } health_report { overall_health: HEALTHY } } }
I20260131 03:12:20.352046 1752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.349958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39698 (local address 127.19.84.2:42987)
0131 03:12:20.350015 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:20.350017 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:20.350025 (+ 8us) server_negotiation.cc:408] Connection header received
0131 03:12:20.350109 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:20.350110 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:20.350128 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:20.350180 (+ 52us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:12:20.350415 (+ 235us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.350884 (+ 469us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.351619 (+ 735us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.351748 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.351784 (+ 36us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:12:20.351867 (+ 83us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:12:20.351928 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:20.351971 (+ 43us) server_negotiation.cc:300] Negotiation successful
0131 03:12:20.351990 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260131 03:12:20.353664 1418 raft_consensus.cc:1275] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Refusing update from remote peer f9f2dc8e777c45fc971651de8f01f690: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:12:20.353665 1690 raft_consensus.cc:1275] T b2306eae776c4627abcb72119604a470 P 2c2bad11ca33403eb47410763aa631e6 [term 1 FOLLOWER]: Refusing update from remote peer f9f2dc8e777c45fc971651de8f01f690: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:12:20.354005 1763 consensus_queue.cc:1048] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:12:20.354127 1760 consensus_queue.cc:1048] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:12:20.356318 1766 mvcc.cc:204] Tried to move back new op lower bound from 7249220158886047744 to 7249220158833565696. Current Snapshot: MvccSnapshot[applied={T|T < 7249220158886047744}]
I20260131 03:12:20.356676 1767 mvcc.cc:204] Tried to move back new op lower bound from 7249220158886047744 to 7249220158833565696. Current Snapshot: MvccSnapshot[applied={T|T < 7249220158886047744}]
I20260131 03:12:20.358918 1270 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:12:20.359001 1270 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:12:20.359930 1270 catalog_manager.cc:5941] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590: Sending DeleteTablet for 3 replicas of tablet b2306eae776c4627abcb72119604a470
I20260131 03:12:20.360286 1398 tablet_service.cc:1552] Processing DeleteTablet for tablet b2306eae776c4627abcb72119604a470 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:12:20 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:34726
I20260131 03:12:20.360348 1670 tablet_service.cc:1552] Processing DeleteTablet for tablet b2306eae776c4627abcb72119604a470 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:12:20 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:40936
I20260131 03:12:20.360337 1534 tablet_service.cc:1552] Processing DeleteTablet for tablet b2306eae776c4627abcb72119604a470 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:12:20 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:39684
I20260131 03:12:20.360570 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 1335
I20260131 03:12:20.362579 1777 tablet_replica.cc:333] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690: stopping tablet replica
I20260131 03:12:20.362659 1777 raft_consensus.cc:2243] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:12:20.362833 1777 raft_consensus.cc:2272] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:12:20.363258 1777 ts_tablet_manager.cc:1916] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:12:20.365381 1777 ts_tablet_manager.cc:1929] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:12:20.365466 1777 log.cc:1199] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/wal/wals/b2306eae776c4627abcb72119604a470
I20260131 03:12:20.365746 1777 ts_tablet_manager.cc:1950] T b2306eae776c4627abcb72119604a470 P f9f2dc8e777c45fc971651de8f01f690: Deleting consensus metadata
I20260131 03:12:20.366158 1255 catalog_manager.cc:4985] TS f9f2dc8e777c45fc971651de8f01f690 (127.19.84.2:42987): tablet b2306eae776c4627abcb72119604a470 (table test-table [id=6954671bacc449418b609c9821288dfb]) successfully deleted
W20260131 03:12:20.367012 1258 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.84.1:35801 (error 108)
I20260131 03:12:20.367254 1778 tablet_replica.cc:333] T b2306eae776c4627abcb72119604a470 P 2c2bad11ca33403eb47410763aa631e6: stopping tablet replica
I20260131 03:12:20.367357 1778 raft_consensus.cc:2243] T b2306eae776c4627abcb72119604a470 P 2c2bad11ca33403eb47410763aa631e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:12:20.367501 1778 raft_consensus.cc:2272] T b2306eae776c4627abcb72119604a470 P 2c2bad11ca33403eb47410763aa631e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:12:20.367551 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.367425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:35801 (local address 127.0.0.1:34740)
0131 03:12:20.367485 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:20.367519 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.1:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260131 03:12:20.367641 1258 catalog_manager.cc:4712] TS 90c0155151a74528816bcf8a9287396d (127.19.84.1:35801): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b2306eae776c4627abcb72119604a470: Network error: Client connection negotiation failed: client connection to 127.19.84.1:35801: connect: Connection refused (error 111)
I20260131 03:12:20.367892 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 1471
I20260131 03:12:20.367982 1778 ts_tablet_manager.cc:1916] T b2306eae776c4627abcb72119604a470 P 2c2bad11ca33403eb47410763aa631e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:12:20.369652 1778 ts_tablet_manager.cc:1929] T b2306eae776c4627abcb72119604a470 P 2c2bad11ca33403eb47410763aa631e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:12:20.369710 1778 log.cc:1199] T b2306eae776c4627abcb72119604a470 P 2c2bad11ca33403eb47410763aa631e6: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/b2306eae776c4627abcb72119604a470
I20260131 03:12:20.369968 1778 ts_tablet_manager.cc:1950] T b2306eae776c4627abcb72119604a470 P 2c2bad11ca33403eb47410763aa631e6: Deleting consensus metadata
I20260131 03:12:20.370359 1255 catalog_manager.cc:4985] TS 2c2bad11ca33403eb47410763aa631e6 (127.19.84.3:39851): tablet b2306eae776c4627abcb72119604a470 (table test-table [id=6954671bacc449418b609c9821288dfb]) successfully deleted
I20260131 03:12:20.373649 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 1607
I20260131 03:12:20.378590 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 1239
I20260131 03:12:20.384250 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:41917
--webserver_interface=127.19.84.62
--webserver_port=32973
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:41917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:12:20.462421 1780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:20.462615 1780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:20.462639 1780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:20.464140 1780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:12:20.464191 1780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:20.464206 1780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:20.464218 1780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:12:20.464231 1780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:12:20.465869 1780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:41917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:41917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=32973
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:20.466115 1780 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:20.466362 1780 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:20.472491 1786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:20.472492 1787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:20.472672 1780 server_base.cc:1060] running on GCE node
W20260131 03:12:20.472519 1789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:20.472957 1780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:20.473278 1780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:20.474442 1780 hybrid_clock.cc:648] HybridClock initialized: now 1769829140474420 us; error 35 us; skew 500 ppm
Jan 31 03:12:20 dist-test-slave-5nnp krb5kdc[1223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829140, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:20.476866 1780 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:12:20.477478 1780 webserver.cc:492] Webserver started at http://127.19.84.62:32973/ using document root <none> and password file <none>
I20260131 03:12:20.477687 1780 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:20.477756 1780 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:20.479015 1780 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:12:20.479735 1796 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:20.479918 1780 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:12:20.480006 1780 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "ec75413d680c4c8c9f4b89582b18d590"
format_stamp: "Formatted at 2026-01-31 03:12:19 on dist-test-slave-5nnp"
server_key: "a5189e440d723a37e47f1e67bd8ac5fe"
server_key_iv: "5121024338d3884ff6c1d3fd147418b7"
server_key_version: "encryptionkey@0"
I20260131 03:12:20.480333 1780 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:20.490479 1780 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:20.492236 1780 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:20.492378 1780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:20.496876 1780 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:41917
I20260131 03:12:20.496994 1848 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:41917 every 8 connection(s)
I20260131 03:12:20.497292 1780 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:12:20.498421 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 1780
I20260131 03:12:20.498533 1849 sys_catalog.cc:263] Verifying existing consensus state
I20260131 03:12:20.498739 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:35801
--local_ip_for_outbound_sockets=127.19.84.1
--tserver_master_addrs=127.19.84.62:41917
--webserver_port=36141
--webserver_interface=127.19.84.1
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260131 03:12:20.499486 1849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590: Bootstrap starting.
I20260131 03:12:20.508147 1849 log.cc:826] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:20.510054 1849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260131 03:12:20.510360 1849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590: Bootstrap complete.
I20260131 03:12:20.513659 1849 raft_consensus.cc:359] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec75413d680c4c8c9f4b89582b18d590" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41917 } }
I20260131 03:12:20.513936 1849 raft_consensus.cc:740] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec75413d680c4c8c9f4b89582b18d590, State: Initialized, Role: FOLLOWER
I20260131 03:12:20.514078 1849 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec75413d680c4c8c9f4b89582b18d590" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41917 } }
I20260131 03:12:20.514140 1849 raft_consensus.cc:399] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:12:20.514212 1849 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:12:20.514261 1849 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 1 FOLLOWER]: Advancing to term 2
I20260131 03:12:20.515084 1849 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec75413d680c4c8c9f4b89582b18d590" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41917 } }
I20260131 03:12:20.515168 1849 leader_election.cc:304] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec75413d680c4c8c9f4b89582b18d590; no voters:
I20260131 03:12:20.515375 1849 leader_election.cc:290] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260131 03:12:20.515443 1853 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 2 FOLLOWER]: Leader election won for term 2
I20260131 03:12:20.515552 1853 raft_consensus.cc:697] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 2 LEADER]: Becoming Leader. State: Replica: ec75413d680c4c8c9f4b89582b18d590, State: Running, Role: LEADER
I20260131 03:12:20.515631 1849 sys_catalog.cc:565] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:12:20.515626 1853 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec75413d680c4c8c9f4b89582b18d590" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41917 } }
I20260131 03:12:20.516573 1854 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ec75413d680c4c8c9f4b89582b18d590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec75413d680c4c8c9f4b89582b18d590" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41917 } } }
I20260131 03:12:20.516601 1855 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec75413d680c4c8c9f4b89582b18d590. Latest consensus state: current_term: 2 leader_uuid: "ec75413d680c4c8c9f4b89582b18d590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec75413d680c4c8c9f4b89582b18d590" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41917 } } }
I20260131 03:12:20.516669 1854 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [sys.catalog]: This master's current role is: LEADER
I20260131 03:12:20.516698 1855 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [sys.catalog]: This master's current role is: LEADER
I20260131 03:12:20.516870 1862 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:12:20.517565 1862 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:12:20.517686 1862 catalog_manager.cc:1269] Loaded cluster ID: 4240b6fddefc4a979e867c316c62a1ee
I20260131 03:12:20.517948 1862 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:12:20.518187 1862 catalog_manager.cc:1324] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260131 03:12:20.521173 1862 catalog_manager.cc:1334] IPKI private key successfully encrypted.
I20260131 03:12:20.523586 1862 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:12:20.523770 1862 catalog_manager.cc:6038] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590: Loaded TSK: 0
I20260131 03:12:20.523969 1862 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260131 03:12:20.583227 1851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:20.583509 1851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:20.583554 1851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:20.585042 1851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:20.585115 1851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:20.585263 1851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:12:20.586822 1851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:35801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=36141
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:41917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:20.587065 1851 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:20.587270 1851 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:20.589787 1879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:20.589792 1876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:20.589792 1877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:20.590376 1851 server_base.cc:1060] running on GCE node
I20260131 03:12:20.590592 1851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:20.590818 1851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:20.591956 1851 hybrid_clock.cc:648] HybridClock initialized: now 1769829140591931 us; error 41 us; skew 500 ppm
Jan 31 03:12:20 dist-test-slave-5nnp krb5kdc[1223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829140, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:20.594408 1851 init.cc:377] Logged in from keytab as kudu/127.19.84.1@KRBTEST.COM (short username kudu)
I20260131 03:12:20.594923 1851 webserver.cc:492] Webserver started at http://127.19.84.1:36141/ using document root <none> and password file <none>
I20260131 03:12:20.595116 1851 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:20.595186 1851 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:20.596525 1851 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:12:20.597184 1886 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:20.597378 1851 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:12:20.597478 1851 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "90c0155151a74528816bcf8a9287396d"
format_stamp: "Formatted at 2026-01-31 03:12:19 on dist-test-slave-5nnp"
server_key: "2e9c122da4dcd1205743b00fa4290596"
server_key_iv: "cfa83252d1cd7a1bd836a91672e9f84a"
server_key_version: "encryptionkey@0"
I20260131 03:12:20.597777 1851 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:20.622653 1851 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:20.624723 1851 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:20.624831 1851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:20.625118 1851 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:12:20.625612 1893 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260131 03:12:20.626525 1851 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260131 03:12:20.626579 1851 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260131 03:12:20.626605 1851 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260131 03:12:20.627195 1851 ts_tablet_manager.cc:616] Registered 1 tablets
I20260131 03:12:20.627245 1851 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260131 03:12:20.627285 1893 tablet_bootstrap.cc:492] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d: Bootstrap starting.
I20260131 03:12:20.635596 1851 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:35801
I20260131 03:12:20.636094 1851 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
I20260131 03:12:20.636679 2000 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:35801 every 8 connection(s)
I20260131 03:12:20.638748 1893 log.cc:826] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:20.640718 1893 tablet_bootstrap.cc:492] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260131 03:12:20.641007 1893 tablet_bootstrap.cc:492] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d: Bootstrap complete.
I20260131 03:12:20.641927 1893 ts_tablet_manager.cc:1403] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260131 03:12:20.642889 1893 raft_consensus.cc:359] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } } peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } }
I20260131 03:12:20.643101 1893 raft_consensus.cc:740] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90c0155151a74528816bcf8a9287396d, State: Initialized, Role: FOLLOWER
I20260131 03:12:20.643213 1893 consensus_queue.cc:260] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } } peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } }
I20260131 03:12:20.643486 1893 ts_tablet_manager.cc:1434] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
Jan 31 03:12:20 dist-test-slave-5nnp krb5kdc[1223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829140, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:12:20.644121 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 1851
I20260131 03:12:20.644408 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:42987
--local_ip_for_outbound_sockets=127.19.84.2
--tserver_master_addrs=127.19.84.62:41917
--webserver_port=34399
--webserver_interface=127.19.84.2
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260131 03:12:20.650602 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.636821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.1:50995)
0131 03:12:20.637315 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:20.637354 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:20.638141 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:20.638607 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:20.638613 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:20.639044 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:20.639346 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.639350 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.640864 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.640866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:20.641411 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.641414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.641871 (+ 457us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.642481 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:20.642504 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:20.646267 (+ 3763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:20.648877 (+ 2610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:20.648883 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:20.648886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:20.649170 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:20.649437 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:20.649438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:20.649439 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:20.649527 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:20.649955 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:20.649961 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:20.650109 (+ 148us) client_negotiation.cc:769] Sending connection context
0131 03:12:20.650236 (+ 127us) client_negotiation.cc:240] Negotiation successful
0131 03:12:20.650395 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":80,"threads_started":1}
I20260131 03:12:20.650744 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.637075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:50995 (local address 127.19.84.62:41917)
0131 03:12:20.637291 (+ 216us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:20.637302 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:20.637475 (+ 173us) server_negotiation.cc:408] Connection header received
0131 03:12:20.638349 (+ 874us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:20.638355 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:20.638406 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:20.638543 (+ 137us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:20.639496 (+ 953us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.640752 (+ 1256us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.641526 (+ 774us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.641749 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.646398 (+ 4649us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:20.646418 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:20.646419 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:20.646444 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:20.648773 (+ 2329us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:20.649267 (+ 494us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:20.649270 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:20.649271 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:20.649346 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:20.649607 (+ 261us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:20.649608 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:20.649609 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:20.649868 (+ 259us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:20.649973 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:20.650331 (+ 358us) server_negotiation.cc:300] Negotiation successful
0131 03:12:20.650490 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":100,"threads_started":1}
I20260131 03:12:20.651501 2001 heartbeater.cc:344] Connected to a master server at 127.19.84.62:41917
I20260131 03:12:20.651612 2001 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:20.651854 2001 heartbeater.cc:507] Master 127.19.84.62:41917 requested a full tablet report, sending...
I20260131 03:12:20.652642 1813 ts_manager.cc:194] Registered new tserver with Master: 90c0155151a74528816bcf8a9287396d (127.19.84.1:35801)
I20260131 03:12:20.653872 1813 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.1@KRBTEST.COM'} at 127.19.84.1:50995
I20260131 03:12:20.654860 2001 heartbeater.cc:499] Master 127.19.84.62:41917 was elected leader, sending a full tablet report...
I20260131 03:12:20.662612 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.653227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:35801 (local address 127.0.0.1:34744)
0131 03:12:20.659519 (+ 6292us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:20.659553 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:20.659647 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:20.659839 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:20.659841 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:20.659844 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:20.660151 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.660154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.661396 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.661397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:20.662316 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.662319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.662397 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.662426 (+ 29us) client_negotiation.cc:769] Sending connection context
0131 03:12:20.662477 (+ 51us) client_negotiation.cc:240] Negotiation successful
0131 03:12:20.662519 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6258,"thread_start_us":96,"threads_started":1}
I20260131 03:12:20.663872 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.654110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34744 (local address 127.19.84.1:35801)
0131 03:12:20.654286 (+ 176us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:20.654294 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:20.659641 (+ 5347us) server_negotiation.cc:408] Connection header received
0131 03:12:20.659718 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:20.659721 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:20.659738 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:20.659812 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:20.660242 (+ 430us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.661294 (+ 1052us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.662413 (+ 1119us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.663686 (+ 1273us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.663728 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:20.663757 (+ 29us) server_negotiation.cc:300] Negotiation successful
0131 03:12:20.663798 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260131 03:12:20.664180 1935 tablet_service.cc:1552] Processing DeleteTablet for tablet b2306eae776c4627abcb72119604a470 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:12:20 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:34744
I20260131 03:12:20.664757 2010 tablet_replica.cc:333] stopping tablet replica
I20260131 03:12:20.664851 2010 raft_consensus.cc:2243] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:12:20.664912 2010 raft_consensus.cc:2272] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:12:20.665195 2010 ts_tablet_manager.cc:1916] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:12:20.667271 2010 ts_tablet_manager.cc:1929] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:12:20.667346 2010 log.cc:1199] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/wal/wals/b2306eae776c4627abcb72119604a470
I20260131 03:12:20.667665 2010 ts_tablet_manager.cc:1950] T b2306eae776c4627abcb72119604a470 P 90c0155151a74528816bcf8a9287396d: Deleting consensus metadata
I20260131 03:12:20.669258 1800 catalog_manager.cc:4985] TS 90c0155151a74528816bcf8a9287396d (127.19.84.1:35801): tablet b2306eae776c4627abcb72119604a470 (table test-table [id=6954671bacc449418b609c9821288dfb]) successfully deleted
W20260131 03:12:20.729928 2007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:20.730125 2007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:20.730150 2007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:20.731781 2007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:20.731834 2007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:20.731863 2007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:12:20.733578 2007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:42987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=34399
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:41917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:20.733831 2007 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:20.734041 2007 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:20.737018 2018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:20.736976 2016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:20.737183 2007 server_base.cc:1060] running on GCE node
W20260131 03:12:20.736972 2015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:20.737447 2007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:20.737635 2007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:20.738806 2007 hybrid_clock.cc:648] HybridClock initialized: now 1769829140738797 us; error 36 us; skew 500 ppm
Jan 31 03:12:20 dist-test-slave-5nnp krb5kdc[1223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829140, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:20.741199 2007 init.cc:377] Logged in from keytab as kudu/127.19.84.2@KRBTEST.COM (short username kudu)
I20260131 03:12:20.741840 2007 webserver.cc:492] Webserver started at http://127.19.84.2:34399/ using document root <none> and password file <none>
I20260131 03:12:20.742048 2007 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:20.742117 2007 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:20.743662 2007 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:12:20.744302 2025 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:20.744671 2007 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:12:20.744791 2007 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "f9f2dc8e777c45fc971651de8f01f690"
format_stamp: "Formatted at 2026-01-31 03:12:20 on dist-test-slave-5nnp"
server_key: "c260a49d134b2599aee18aacbcca8528"
server_key_iv: "95cfd6c7349fb3d6780b261f08408e1a"
server_key_version: "encryptionkey@0"
I20260131 03:12:20.745112 2007 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:20.758175 2007 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:20.760290 2007 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:20.760469 2007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:20.760732 2007 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:12:20.761044 2007 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:12:20.761097 2007 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:20.761142 2007 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:12:20.761173 2007 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:20.767326 2007 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:42987
I20260131 03:12:20.767436 2138 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:42987 every 8 connection(s)
I20260131 03:12:20.767767 2007 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
I20260131 03:12:20.769631 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 2007
I20260131 03:12:20.769945 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:39851
--local_ip_for_outbound_sockets=127.19.84.3
--tserver_master_addrs=127.19.84.62:41917
--webserver_port=38147
--webserver_interface=127.19.84.3
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jan 31 03:12:20 dist-test-slave-5nnp krb5kdc[1223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829140, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:12:20.777637 2141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.768525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.2:33947)
0131 03:12:20.768705 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:20.768733 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:20.769218 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:20.769600 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:20.769603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:20.769966 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:20.770349 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.770353 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.771331 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.771332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:20.771987 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.771989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.772098 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.773030 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:20.773046 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:20.774458 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:20.776435 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:20.776441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:20.776444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:20.776639 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:20.776816 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:20.776817 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:20.776817 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:20.776885 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:20.777158 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:20.777162 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:20.777280 (+ 118us) client_negotiation.cc:769] Sending connection context
0131 03:12:20.777374 (+ 94us) client_negotiation.cc:240] Negotiation successful
0131 03:12:20.777494 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":107,"threads_started":1}
I20260131 03:12:20.777675 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.768551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:33947 (local address 127.19.84.62:41917)
0131 03:12:20.768632 (+ 81us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:20.768635 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:20.768817 (+ 182us) server_negotiation.cc:408] Connection header received
0131 03:12:20.769319 (+ 502us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:20.769321 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:20.769466 (+ 145us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:20.769543 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:20.770456 (+ 913us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.771224 (+ 768us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.772094 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.772313 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.774558 (+ 2245us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:20.774565 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:20.774566 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:20.774575 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:20.776346 (+ 1771us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:20.776716 (+ 370us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:20.776717 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:20.776718 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:20.776746 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:20.776950 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:20.776951 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:20.776951 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:20.777089 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:20.777162 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:20.777422 (+ 260us) server_negotiation.cc:300] Negotiation successful
0131 03:12:20.777531 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260131 03:12:20.778170 2139 heartbeater.cc:344] Connected to a master server at 127.19.84.62:41917
I20260131 03:12:20.778294 2139 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:20.778538 2139 heartbeater.cc:507] Master 127.19.84.62:41917 requested a full tablet report, sending...
I20260131 03:12:20.779031 1813 ts_manager.cc:194] Registered new tserver with Master: f9f2dc8e777c45fc971651de8f01f690 (127.19.84.2:42987)
I20260131 03:12:20.779510 1813 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.2@KRBTEST.COM'} at 127.19.84.2:33947
W20260131 03:12:20.851543 2142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:20.851749 2142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:20.851784 2142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:20.853246 2142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:20.853318 2142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:20.853361 2142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:12:20.854959 2142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:39851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=38147
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:41917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:20.855199 2142 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:20.855435 2142 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:20.858007 2148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:20.858022 2142 server_base.cc:1060] running on GCE node
W20260131 03:12:20.858008 2147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:20.858006 2150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:20.858434 2142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:20.858656 2142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:20.859813 2142 hybrid_clock.cc:648] HybridClock initialized: now 1769829140859800 us; error 26 us; skew 500 ppm
Jan 31 03:12:20 dist-test-slave-5nnp krb5kdc[1223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829140, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:20.862151 2142 init.cc:377] Logged in from keytab as kudu/127.19.84.3@KRBTEST.COM (short username kudu)
I20260131 03:12:20.862777 2142 webserver.cc:492] Webserver started at http://127.19.84.3:38147/ using document root <none> and password file <none>
I20260131 03:12:20.862974 2142 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:20.863045 2142 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:20.864348 2142 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:12:20.864915 2157 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:20.865131 2142 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260131 03:12:20.865226 2142 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "2c2bad11ca33403eb47410763aa631e6"
format_stamp: "Formatted at 2026-01-31 03:12:20 on dist-test-slave-5nnp"
server_key: "355a3d381a466f666c5140d09ec65643"
server_key_iv: "2cca9d643fb5fdb2cd070cc33caff6e2"
server_key_version: "encryptionkey@0"
I20260131 03:12:20.865507 2142 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:20.881667 2142 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:20.883505 2142 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:20.883654 2142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:20.883882 2142 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:12:20.884254 2142 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:12:20.884311 2142 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:20.884356 2142 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:12:20.884383 2142 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:20.890393 2142 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:39851
I20260131 03:12:20.890436 2270 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:39851 every 8 connection(s)
I20260131 03:12:20.890750 2142 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
I20260131 03:12:20.895061 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 2142
Jan 31 03:12:20 dist-test-slave-5nnp krb5kdc[1223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829140, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:12:20.899667 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.891489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:39527 (local address 127.19.84.62:41917)
0131 03:12:20.891538 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:20.891540 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:20.891704 (+ 164us) server_negotiation.cc:408] Connection header received
0131 03:12:20.892158 (+ 454us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:20.892159 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:20.892168 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:20.892230 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:20.892949 (+ 719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.893432 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.894024 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.894145 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.896789 (+ 2644us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:20.896793 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:20.896793 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:20.896800 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:20.898295 (+ 1495us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:20.898694 (+ 399us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:20.898695 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:20.898696 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:20.898722 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:20.898940 (+ 218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:20.898940 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:20.898941 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:20.899028 (+ 87us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:20.899102 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:20.899467 (+ 365us) server_negotiation.cc:300] Negotiation successful
0131 03:12:20.899555 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260131 03:12:20.899688 2273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.891442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.3:39527)
0131 03:12:20.891581 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:20.891613 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:20.892083 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:20.892288 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:20.892292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:20.892594 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:20.892871 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.892874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.893538 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.893539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:20.893937 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.893939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.894052 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.895053 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:20.895078 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:20.896712 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:20.898391 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:20.898397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:20.898400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:20.898606 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:20.898791 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:20.898792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:20.898793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:20.898873 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:20.899092 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:20.899098 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:20.899263 (+ 165us) client_negotiation.cc:769] Sending connection context
0131 03:12:20.899438 (+ 175us) client_negotiation.cc:240] Negotiation successful
0131 03:12:20.899551 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":79,"threads_started":1}
I20260131 03:12:20.900249 2271 heartbeater.cc:344] Connected to a master server at 127.19.84.62:41917
I20260131 03:12:20.900343 2271 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:20.900559 2271 heartbeater.cc:507] Master 127.19.84.62:41917 requested a full tablet report, sending...
I20260131 03:12:20.900990 1813 ts_manager.cc:194] Registered new tserver with Master: 2c2bad11ca33403eb47410763aa631e6 (127.19.84.3:39851)
I20260131 03:12:20.901403 1813 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:39527
I20260131 03:12:20.902136 2275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.895550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58938 (local address 127.19.84.62:41917)
0131 03:12:20.895672 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:20.895673 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:20.895682 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:12:20.895757 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:20.895758 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:20.895769 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:20.895849 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:20.896571 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.897277 (+ 706us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.897974 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.898163 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.898963 (+ 800us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:20.898967 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:20.898967 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:20.898974 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:20.900568 (+ 1594us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:20.900933 (+ 365us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:20.900934 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:20.900935 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:20.900964 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:20.901145 (+ 181us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:20.901146 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:20.901146 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:20.901235 (+ 89us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:20.901383 (+ 148us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:20.901506 (+ 123us) server_negotiation.cc:300] Negotiation successful
0131 03:12:20.901600 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260131 03:12:20.903033 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:12:20.905474 2275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.903431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58954 (local address 127.19.84.62:41917)
0131 03:12:20.903491 (+ 60us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:20.903493 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:20.903604 (+ 111us) server_negotiation.cc:408] Connection header received
0131 03:12:20.903640 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:20.903640 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:20.903663 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:20.903727 (+ 64us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:12:20.904055 (+ 328us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.904524 (+ 469us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.905049 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.905152 (+ 103us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.905182 (+ 30us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:12:20.905283 (+ 101us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:12:20.905342 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:20.905403 (+ 61us) server_negotiation.cc:300] Negotiation successful
0131 03:12:20.905425 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260131 03:12:20.905742 1813 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:58954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:12:20.906996 1813 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:12:20.909602 1935 tablet_service.cc:1505] Processing CreateTablet for tablet 4748055821b043e49b2a947a6718824f (DEFAULT_TABLE table=test-table [id=de56e9adc27f49b59e8c9edadcfa0385]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:12:20.909832 1935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4748055821b043e49b2a947a6718824f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:12:20.911391 1893 tablet_bootstrap.cc:492] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d: Bootstrap starting.
I20260131 03:12:20.911705 1893 tablet_bootstrap.cc:654] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d: Neither blocks nor log segments found. Creating new log.
I20260131 03:12:20.912330 1893 tablet_bootstrap.cc:492] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d: No bootstrap required, opened a new log
I20260131 03:12:20.912393 1893 ts_tablet_manager.cc:1403] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260131 03:12:20.912453 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.909068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:42987 (local address 127.0.0.1:39704)
0131 03:12:20.909125 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:20.909138 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:20.909174 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:20.909789 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:20.909790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:20.909793 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:20.909971 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.909974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.911394 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.911394 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:20.912232 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.912234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.912297 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.912302 (+ 5us) client_negotiation.cc:769] Sending connection context
0131 03:12:20.912333 (+ 31us) client_negotiation.cc:240] Negotiation successful
0131 03:12:20.912367 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260131 03:12:20.912516 1893 raft_consensus.cc:359] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } } peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } }
I20260131 03:12:20.912587 1893 raft_consensus.cc:385] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:12:20.912621 1893 raft_consensus.cc:740] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90c0155151a74528816bcf8a9287396d, State: Initialized, Role: FOLLOWER
I20260131 03:12:20.912665 1893 consensus_queue.cc:260] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } } peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } }
I20260131 03:12:20.912737 2277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.909227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:39851 (local address 127.0.0.1:40944)
0131 03:12:20.909407 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:20.909417 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:20.909496 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:20.909900 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:20.909901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:20.909903 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:20.910123 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.910125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.911077 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.911078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:20.912538 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:20.912540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.912604 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.912608 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:12:20.912639 (+ 31us) client_negotiation.cc:240] Negotiation successful
0131 03:12:20.912681 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":130,"threads_started":1}
I20260131 03:12:20.912750 1893 ts_tablet_manager.cc:1434] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:20.913343 2278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.909213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39704 (local address 127.19.84.2:42987)
0131 03:12:20.909616 (+ 403us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:20.909624 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:20.909635 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:12:20.909675 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:20.909677 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:20.909696 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:20.910055 (+ 359us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:20.910152 (+ 97us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.911249 (+ 1097us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.912577 (+ 1328us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.913136 (+ 559us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.913192 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:20.913224 (+ 32us) server_negotiation.cc:300] Negotiation successful
0131 03:12:20.913265 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":163,"threads_started":1}
I20260131 03:12:20.913542 2279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:20.909518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40944 (local address 127.19.84.3:39851)
0131 03:12:20.909717 (+ 199us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:20.909726 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:20.909739 (+ 13us) server_negotiation.cc:408] Connection header received
0131 03:12:20.909778 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:20.909780 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:20.909796 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:20.910219 (+ 423us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:20.910312 (+ 93us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.910989 (+ 677us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:20.912831 (+ 1842us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:20.913358 (+ 527us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:20.913397 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:20.913426 (+ 29us) server_negotiation.cc:300] Negotiation successful
0131 03:12:20.913472 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":110,"threads_started":1}
I20260131 03:12:20.913910 2065 tablet_service.cc:1505] Processing CreateTablet for tablet 4748055821b043e49b2a947a6718824f (DEFAULT_TABLE table=test-table [id=de56e9adc27f49b59e8c9edadcfa0385]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:12:20.914148 2065 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4748055821b043e49b2a947a6718824f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:12:20.914161 2205 tablet_service.cc:1505] Processing CreateTablet for tablet 4748055821b043e49b2a947a6718824f (DEFAULT_TABLE table=test-table [id=de56e9adc27f49b59e8c9edadcfa0385]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:12:20.914462 2205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4748055821b043e49b2a947a6718824f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:12:20.916607 2281 tablet_bootstrap.cc:492] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690: Bootstrap starting.
I20260131 03:12:20.916723 2282 tablet_bootstrap.cc:492] T 4748055821b043e49b2a947a6718824f P 2c2bad11ca33403eb47410763aa631e6: Bootstrap starting.
I20260131 03:12:20.917254 2281 tablet_bootstrap.cc:654] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690: Neither blocks nor log segments found. Creating new log.
I20260131 03:12:20.917480 2282 tablet_bootstrap.cc:654] T 4748055821b043e49b2a947a6718824f P 2c2bad11ca33403eb47410763aa631e6: Neither blocks nor log segments found. Creating new log.
I20260131 03:12:20.917578 2281 log.cc:826] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:20.917788 2282 log.cc:826] T 4748055821b043e49b2a947a6718824f P 2c2bad11ca33403eb47410763aa631e6: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:20.918387 2281 tablet_bootstrap.cc:492] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690: No bootstrap required, opened a new log
I20260131 03:12:20.918474 2282 tablet_bootstrap.cc:492] T 4748055821b043e49b2a947a6718824f P 2c2bad11ca33403eb47410763aa631e6: No bootstrap required, opened a new log
I20260131 03:12:20.918469 2281 ts_tablet_manager.cc:1403] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:12:20.918537 2282 ts_tablet_manager.cc:1403] T 4748055821b043e49b2a947a6718824f P 2c2bad11ca33403eb47410763aa631e6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:12:20.920076 2281 raft_consensus.cc:359] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } } peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } }
I20260131 03:12:20.920076 2282 raft_consensus.cc:359] T 4748055821b043e49b2a947a6718824f P 2c2bad11ca33403eb47410763aa631e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } } peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } }
I20260131 03:12:20.920233 2281 raft_consensus.cc:385] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:12:20.920240 2282 raft_consensus.cc:385] T 4748055821b043e49b2a947a6718824f P 2c2bad11ca33403eb47410763aa631e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:12:20.920269 2281 raft_consensus.cc:740] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9f2dc8e777c45fc971651de8f01f690, State: Initialized, Role: FOLLOWER
I20260131 03:12:20.920269 2282 raft_consensus.cc:740] T 4748055821b043e49b2a947a6718824f P 2c2bad11ca33403eb47410763aa631e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c2bad11ca33403eb47410763aa631e6, State: Initialized, Role: FOLLOWER
I20260131 03:12:20.920358 2281 consensus_queue.cc:260] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } } peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } }
I20260131 03:12:20.920358 2282 consensus_queue.cc:260] T 4748055821b043e49b2a947a6718824f P 2c2bad11ca33403eb47410763aa631e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } } peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } }
I20260131 03:12:20.920615 2281 ts_tablet_manager.cc:1434] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:12:20.920615 2282 ts_tablet_manager.cc:1434] T 4748055821b043e49b2a947a6718824f P 2c2bad11ca33403eb47410763aa631e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:12:20.920677 2139 heartbeater.cc:499] Master 127.19.84.62:41917 was elected leader, sending a full tablet report...
I20260131 03:12:20.920677 2271 heartbeater.cc:499] Master 127.19.84.62:41917 was elected leader, sending a full tablet report...
W20260131 03:12:21.018528 2140 tablet.cc:2378] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:12:21.080984 2285 raft_consensus.cc:493] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:12:21.081122 2285 raft_consensus.cc:515] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } } peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } }
I20260131 03:12:21.081418 2285 leader_election.cc:290] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c2bad11ca33403eb47410763aa631e6 (127.19.84.3:39851), 90c0155151a74528816bcf8a9287396d (127.19.84.1:35801)
I20260131 03:12:21.084025 2141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:21.081618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:39851 (local address 127.19.84.2:36623)
0131 03:12:21.081678 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:21.081692 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:21.081760 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:21.081924 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:21.081924 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:21.081926 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:21.082120 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:21.082121 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:21.083053 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:21.083054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:21.083783 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:21.083785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:21.083876 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:21.083880 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:12:21.083912 (+ 32us) client_negotiation.cc:240] Negotiation successful
0131 03:12:21.083952 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260131 03:12:21.084229 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:21.081618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:35801 (local address 127.19.84.2:54895)
0131 03:12:21.081884 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:21.081893 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:21.081976 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:21.082148 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:21.082149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:21.082150 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:21.082395 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:21.082396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:21.083254 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:21.083255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:21.083962 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:21.083964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:21.084054 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:21.084057 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:12:21.084090 (+ 33us) client_negotiation.cc:240] Negotiation successful
0131 03:12:21.084125 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":89,"thread_start_us":103,"threads_started":1}
I20260131 03:12:21.084589 2279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:21.081739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:36623 (local address 127.19.84.3:39851)
0131 03:12:21.081784 (+ 45us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:21.081787 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:21.081799 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:12:21.081829 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:21.081830 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:21.081842 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:21.081921 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:21.082198 (+ 277us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:21.082951 (+ 753us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:21.083875 (+ 924us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:21.084490 (+ 615us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:21.084498 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:21.084519 (+ 21us) server_negotiation.cc:300] Negotiation successful
0131 03:12:21.084542 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260131 03:12:21.084599 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:21.081723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:54895 (local address 127.19.84.1:35801)
0131 03:12:21.081773 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:21.081776 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:21.081960 (+ 184us) server_negotiation.cc:408] Connection header received
0131 03:12:21.082052 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:21.082053 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:21.082062 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:21.082131 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:21.082484 (+ 353us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:21.082951 (+ 467us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:21.084055 (+ 1104us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:21.084504 (+ 449us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:21.084511 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:21.084530 (+ 19us) server_negotiation.cc:300] Negotiation successful
0131 03:12:21.084554 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260131 03:12:21.084882 2225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4748055821b043e49b2a947a6718824f" candidate_uuid: "f9f2dc8e777c45fc971651de8f01f690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c2bad11ca33403eb47410763aa631e6" is_pre_election: true
I20260131 03:12:21.084882 1943 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4748055821b043e49b2a947a6718824f" candidate_uuid: "f9f2dc8e777c45fc971651de8f01f690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c0155151a74528816bcf8a9287396d" is_pre_election: true
I20260131 03:12:21.085057 2225 raft_consensus.cc:2468] T 4748055821b043e49b2a947a6718824f P 2c2bad11ca33403eb47410763aa631e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9f2dc8e777c45fc971651de8f01f690 in term 0.
I20260131 03:12:21.085062 1943 raft_consensus.cc:2468] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9f2dc8e777c45fc971651de8f01f690 in term 0.
I20260131 03:12:21.085307 2026 leader_election.cc:304] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c2bad11ca33403eb47410763aa631e6, f9f2dc8e777c45fc971651de8f01f690; no voters:
I20260131 03:12:21.085438 2285 raft_consensus.cc:2804] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:12:21.085498 2285 raft_consensus.cc:493] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:12:21.085533 2285 raft_consensus.cc:3060] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:12:21.086167 2285 raft_consensus.cc:515] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } } peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } }
I20260131 03:12:21.086329 2285 leader_election.cc:290] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690 [CANDIDATE]: Term 1 election: Requested vote from peers 2c2bad11ca33403eb47410763aa631e6 (127.19.84.3:39851), 90c0155151a74528816bcf8a9287396d (127.19.84.1:35801)
I20260131 03:12:21.086522 2225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4748055821b043e49b2a947a6718824f" candidate_uuid: "f9f2dc8e777c45fc971651de8f01f690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c2bad11ca33403eb47410763aa631e6"
I20260131 03:12:21.086521 1943 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4748055821b043e49b2a947a6718824f" candidate_uuid: "f9f2dc8e777c45fc971651de8f01f690" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c0155151a74528816bcf8a9287396d"
I20260131 03:12:21.086611 2225 raft_consensus.cc:3060] T 4748055821b043e49b2a947a6718824f P 2c2bad11ca33403eb47410763aa631e6 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:12:21.086607 1943 raft_consensus.cc:3060] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:12:21.087215 1943 raft_consensus.cc:2468] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9f2dc8e777c45fc971651de8f01f690 in term 1.
I20260131 03:12:21.087430 2225 raft_consensus.cc:2468] T 4748055821b043e49b2a947a6718824f P 2c2bad11ca33403eb47410763aa631e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9f2dc8e777c45fc971651de8f01f690 in term 1.
I20260131 03:12:21.087432 2029 leader_election.cc:304] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90c0155151a74528816bcf8a9287396d, f9f2dc8e777c45fc971651de8f01f690; no voters:
I20260131 03:12:21.087538 2285 raft_consensus.cc:2804] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:12:21.087679 2285 raft_consensus.cc:697] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690 [term 1 LEADER]: Becoming Leader. State: Replica: f9f2dc8e777c45fc971651de8f01f690, State: Running, Role: LEADER
I20260131 03:12:21.087791 2285 consensus_queue.cc:237] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } } peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } }
I20260131 03:12:21.088619 1812 catalog_manager.cc:5654] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9f2dc8e777c45fc971651de8f01f690 (127.19.84.2). New cstate: current_term: 1 leader_uuid: "f9f2dc8e777c45fc971651de8f01f690" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } health_report { overall_health: UNKNOWN } } }
I20260131 03:12:21.096182 2278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:21.094029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39718 (local address 127.19.84.2:42987)
0131 03:12:21.094086 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:21.094088 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:21.094097 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:12:21.094154 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:21.094155 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:21.094174 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:21.094238 (+ 64us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:12:21.094469 (+ 231us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:21.094930 (+ 461us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:21.095617 (+ 687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:21.095781 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:21.095816 (+ 35us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:12:21.095903 (+ 87us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:12:21.095966 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:21.096015 (+ 49us) server_negotiation.cc:300] Negotiation successful
0131 03:12:21.096034 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260131 03:12:21.097790 1943 raft_consensus.cc:1275] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Refusing update from remote peer f9f2dc8e777c45fc971651de8f01f690: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:12:21.097803 2225 raft_consensus.cc:1275] T 4748055821b043e49b2a947a6718824f P 2c2bad11ca33403eb47410763aa631e6 [term 1 FOLLOWER]: Refusing update from remote peer f9f2dc8e777c45fc971651de8f01f690: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:12:21.098093 2288 consensus_queue.cc:1048] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:12:21.098204 2285 consensus_queue.cc:1048] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:12:21.100396 2291 mvcc.cc:204] Tried to move back new op lower bound from 7249220161934069760 to 7249220161896296448. Current Snapshot: MvccSnapshot[applied={T|T < 7249220161934069760}]
I20260131 03:12:21.102651 1812 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:58954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:12:21.102748 1812 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:58954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:12:21.103470 1812 catalog_manager.cc:5941] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590: Sending DeleteTablet for 3 replicas of tablet 4748055821b043e49b2a947a6718824f
I20260131 03:12:21.103782 2065 tablet_service.cc:1552] Processing DeleteTablet for tablet 4748055821b043e49b2a947a6718824f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:12:21 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:39704
I20260131 03:12:21.103780 2205 tablet_service.cc:1552] Processing DeleteTablet for tablet 4748055821b043e49b2a947a6718824f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:12:21 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:40944
I20260131 03:12:21.103984 1935 tablet_service.cc:1552] Processing DeleteTablet for tablet 4748055821b043e49b2a947a6718824f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:12:21 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:34744
I20260131 03:12:21.104030 2301 tablet_replica.cc:333] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690: stopping tablet replica
I20260131 03:12:21.104040 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 1851
I20260131 03:12:21.104065 2010 tablet_replica.cc:333] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d: stopping tablet replica
I20260131 03:12:21.104105 2010 raft_consensus.cc:2243] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:12:21.104110 2301 raft_consensus.cc:2243] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:12:21.104146 2010 raft_consensus.cc:2272] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:12:21.104200 2301 raft_consensus.cc:2272] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:12:21.105652 2301 ts_tablet_manager.cc:1916] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:12:21.107519 2302 tablet_replica.cc:333] T 4748055821b043e49b2a947a6718824f P 2c2bad11ca33403eb47410763aa631e6: stopping tablet replica
I20260131 03:12:21.107537 2301 ts_tablet_manager.cc:1929] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:12:21.107621 2302 raft_consensus.cc:2243] T 4748055821b043e49b2a947a6718824f P 2c2bad11ca33403eb47410763aa631e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:12:21.107718 2302 raft_consensus.cc:2272] T 4748055821b043e49b2a947a6718824f P 2c2bad11ca33403eb47410763aa631e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:12:21.107729 2301 log.cc:1199] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/wal/wals/4748055821b043e49b2a947a6718824f
I20260131 03:12:21.108146 2301 ts_tablet_manager.cc:1950] T 4748055821b043e49b2a947a6718824f P f9f2dc8e777c45fc971651de8f01f690: Deleting consensus metadata
I20260131 03:12:21.108611 1797 catalog_manager.cc:4985] TS f9f2dc8e777c45fc971651de8f01f690 (127.19.84.2:42987): tablet 4748055821b043e49b2a947a6718824f (table test-table [id=de56e9adc27f49b59e8c9edadcfa0385]) successfully deleted
W20260131 03:12:21.109529 1800 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.84.1:35801 (error 108)
I20260131 03:12:21.110023 2277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:21.109886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:35801 (local address 127.0.0.1:34760)
0131 03:12:21.109945 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:21.109988 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.1:35801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260131 03:12:21.110095 2302 ts_tablet_manager.cc:1916] T 4748055821b043e49b2a947a6718824f P 2c2bad11ca33403eb47410763aa631e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:12:21.110146 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 2007
W20260131 03:12:21.110250 1800 catalog_manager.cc:4712] TS 90c0155151a74528816bcf8a9287396d (127.19.84.1:35801): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4748055821b043e49b2a947a6718824f: Network error: Client connection negotiation failed: client connection to 127.19.84.1:35801: connect: Connection refused (error 111)
I20260131 03:12:21.112550 2302 ts_tablet_manager.cc:1929] T 4748055821b043e49b2a947a6718824f P 2c2bad11ca33403eb47410763aa631e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:12:21.112635 2302 log.cc:1199] T 4748055821b043e49b2a947a6718824f P 2c2bad11ca33403eb47410763aa631e6: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/4748055821b043e49b2a947a6718824f
I20260131 03:12:21.112928 2302 ts_tablet_manager.cc:1950] T 4748055821b043e49b2a947a6718824f P 2c2bad11ca33403eb47410763aa631e6: Deleting consensus metadata
I20260131 03:12:21.113386 1797 catalog_manager.cc:4985] TS 2c2bad11ca33403eb47410763aa631e6 (127.19.84.3:39851): tablet 4748055821b043e49b2a947a6718824f (table test-table [id=de56e9adc27f49b59e8c9edadcfa0385]) successfully deleted
I20260131 03:12:21.115940 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 2142
I20260131 03:12:21.120465 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 1780
I20260131 03:12:21.125757 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:41917
--webserver_interface=127.19.84.62
--webserver_port=32973
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:41917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:12:21.205482 2304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:21.205735 2304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:21.205776 2304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:21.207297 2304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:12:21.207439 2304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:21.207484 2304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:21.207523 2304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:12:21.207551 2304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:12:21.209210 2304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:41917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:41917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=32973
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:21.209494 2304 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:21.209689 2304 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:21.212175 2309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:21.212312 2304 server_base.cc:1060] running on GCE node
W20260131 03:12:21.212198 2310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:21.212237 2312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:21.212622 2304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:21.212874 2304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:21.214026 2304 hybrid_clock.cc:648] HybridClock initialized: now 1769829141214031 us; error 44 us; skew 500 ppm
Jan 31 03:12:21 dist-test-slave-5nnp krb5kdc[1223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829141, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:21.216686 2304 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:12:21.217278 2304 webserver.cc:492] Webserver started at http://127.19.84.62:32973/ using document root <none> and password file <none>
I20260131 03:12:21.217504 2304 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:21.217561 2304 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:21.218784 2304 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:12:21.219485 2319 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:21.219660 2304 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:12:21.219758 2304 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "ec75413d680c4c8c9f4b89582b18d590"
format_stamp: "Formatted at 2026-01-31 03:12:19 on dist-test-slave-5nnp"
server_key: "a5189e440d723a37e47f1e67bd8ac5fe"
server_key_iv: "5121024338d3884ff6c1d3fd147418b7"
server_key_version: "encryptionkey@0"
I20260131 03:12:21.220036 2304 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:21.236325 2304 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:21.238570 2304 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:21.238689 2304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:21.243294 2304 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:41917
I20260131 03:12:21.243435 2371 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:41917 every 8 connection(s)
I20260131 03:12:21.243876 2304 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:12:21.245169 2372 sys_catalog.cc:263] Verifying existing consensus state
I20260131 03:12:21.245704 2372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590: Bootstrap starting.
I20260131 03:12:21.250352 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 2304
I20260131 03:12:21.250654 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:35801
--local_ip_for_outbound_sockets=127.19.84.1
--tserver_master_addrs=127.19.84.62:41917
--webserver_port=36141
--webserver_interface=127.19.84.1
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260131 03:12:21.254308 2372 log.cc:826] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:21.256703 2372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260131 03:12:21.256994 2372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590: Bootstrap complete.
I20260131 03:12:21.258991 2372 raft_consensus.cc:359] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec75413d680c4c8c9f4b89582b18d590" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41917 } }
I20260131 03:12:21.259198 2372 raft_consensus.cc:740] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec75413d680c4c8c9f4b89582b18d590, State: Initialized, Role: FOLLOWER
I20260131 03:12:21.259368 2372 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec75413d680c4c8c9f4b89582b18d590" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41917 } }
I20260131 03:12:21.259469 2372 raft_consensus.cc:399] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:12:21.259517 2372 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:12:21.259572 2372 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 2 FOLLOWER]: Advancing to term 3
I20260131 03:12:21.260452 2372 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec75413d680c4c8c9f4b89582b18d590" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41917 } }
I20260131 03:12:21.260576 2372 leader_election.cc:304] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec75413d680c4c8c9f4b89582b18d590; no voters:
I20260131 03:12:21.260754 2372 leader_election.cc:290] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260131 03:12:21.260821 2376 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 3 FOLLOWER]: Leader election won for term 3
I20260131 03:12:21.260964 2376 raft_consensus.cc:697] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [term 3 LEADER]: Becoming Leader. State: Replica: ec75413d680c4c8c9f4b89582b18d590, State: Running, Role: LEADER
I20260131 03:12:21.261010 2372 sys_catalog.cc:565] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:12:21.261050 2376 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec75413d680c4c8c9f4b89582b18d590" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41917 } }
I20260131 03:12:21.261310 2376 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec75413d680c4c8c9f4b89582b18d590. Latest consensus state: current_term: 3 leader_uuid: "ec75413d680c4c8c9f4b89582b18d590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec75413d680c4c8c9f4b89582b18d590" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41917 } } }
I20260131 03:12:21.261400 2376 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [sys.catalog]: This master's current role is: LEADER
I20260131 03:12:21.261312 2377 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ec75413d680c4c8c9f4b89582b18d590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec75413d680c4c8c9f4b89582b18d590" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41917 } } }
I20260131 03:12:21.261744 2377 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec75413d680c4c8c9f4b89582b18d590 [sys.catalog]: This master's current role is: LEADER
I20260131 03:12:21.261816 2380 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:12:21.262576 2380 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:12:21.262682 2380 catalog_manager.cc:1269] Loaded cluster ID: 4240b6fddefc4a979e867c316c62a1ee
I20260131 03:12:21.262722 2380 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
F20260131 03:12:21.262910 2380 catalog_manager.cc:1476] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fabba22adcd google::LogMessage::Fail() at ??:0
@ 0x7fabba22eb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fabba22a7cc google::LogMessage::Flush() at ??:0
@ 0x7fabba22bf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fabbca7ba66 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fabbca861c4 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fabbb32c830 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fabbb322907 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fabbb6576db start_thread at ??:0
@ 0x7fabb977f71f clone at ??:0
W20260131 03:12:21.346489 2374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:21.346696 2374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:21.346732 2374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:21.348233 2374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:21.348305 2374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:21.348352 2374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:12:21.349969 2374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:35801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=36141
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:41917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:21.350239 2374 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:21.350433 2374 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:21.353437 2409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:21.353457 2408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:21.353761 2374 server_base.cc:1060] running on GCE node
W20260131 03:12:21.353437 2411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:21.354000 2374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:21.354246 2374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:21.355441 2374 hybrid_clock.cc:648] HybridClock initialized: now 1769829141355410 us; error 40 us; skew 500 ppm
Jan 31 03:12:21 dist-test-slave-5nnp krb5kdc[1223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829141, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:21.357896 2374 init.cc:377] Logged in from keytab as kudu/127.19.84.1@KRBTEST.COM (short username kudu)
I20260131 03:12:21.358459 2374 webserver.cc:492] Webserver started at http://127.19.84.1:36141/ using document root <none> and password file <none>
I20260131 03:12:21.358661 2374 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:21.358717 2374 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:21.360117 2374 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:12:21.360836 2418 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:21.361234 2374 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:12:21.361331 2374 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "90c0155151a74528816bcf8a9287396d"
format_stamp: "Formatted at 2026-01-31 03:12:19 on dist-test-slave-5nnp"
server_key: "2e9c122da4dcd1205743b00fa4290596"
server_key_iv: "cfa83252d1cd7a1bd836a91672e9f84a"
server_key_version: "encryptionkey@0"
I20260131 03:12:21.361644 2374 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:21.373598 2374 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:21.375905 2374 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:21.376053 2374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:21.376292 2374 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:12:21.376761 2425 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260131 03:12:21.377955 2374 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260131 03:12:21.378127 2374 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260131 03:12:21.378207 2374 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260131 03:12:21.378779 2374 ts_tablet_manager.cc:616] Registered 1 tablets
I20260131 03:12:21.378844 2374 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260131 03:12:21.378901 2425 tablet_bootstrap.cc:492] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d: Bootstrap starting.
I20260131 03:12:21.386574 2374 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:35801
I20260131 03:12:21.386663 2532 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:35801 every 8 connection(s)
I20260131 03:12:21.387015 2374 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
I20260131 03:12:21.389374 2425 log.cc:826] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:21.391185 2425 tablet_bootstrap.cc:492] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260131 03:12:21.391523 2425 tablet_bootstrap.cc:492] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d: Bootstrap complete.
I20260131 03:12:21.392445 2425 ts_tablet_manager.cc:1403] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.010s
I20260131 03:12:21.393268 2425 raft_consensus.cc:359] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } } peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } }
I20260131 03:12:21.393478 2425 raft_consensus.cc:740] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90c0155151a74528816bcf8a9287396d, State: Initialized, Role: FOLLOWER
I20260131 03:12:21.393617 2425 consensus_queue.cc:260] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } } peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } }
I20260131 03:12:21.393875 2425 ts_tablet_manager.cc:1434] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260131 03:12:21.395434 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 2374
I20260131 03:12:21.395711 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:42987
--local_ip_for_outbound_sockets=127.19.84.2
--tserver_master_addrs=127.19.84.62:41917
--webserver_port=34399
--webserver_interface=127.19.84.2
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:12:21.472662 2535 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:12:21.387926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.1:56571)
0131 03:12:21.388112 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:21.388149 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:21.388893 (+ 744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:21.472502 (+ 83609us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
W20260131 03:12:21.472978 2533 heartbeater.cc:646] Failed to heartbeat to 127.19.84.62:41917 (0 consecutive failures): Network error: Failed to ping master at 127.19.84.62:41917: Client connection negotiation failed: client connection to 127.19.84.62:41917: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260131 03:12:21.473369 2535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:21.473232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.1:40671)
0131 03:12:21.473295 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:21.473340 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260131 03:12:21.473722 2535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:21.473611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.1:32779)
0131 03:12:21.473663 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:21.473694 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260131 03:12:21.473847 2533 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260131 03:12:21.475950 2538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:21.476114 2538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:21.476166 2538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:21.477676 2538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:21.477720 2538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:21.477747 2538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:12:21.479506 2538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:42987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=34399
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:41917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:21.479748 2538 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:21.479939 2538 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:21.482746 2543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:21.482792 2538 server_base.cc:1060] running on GCE node
W20260131 03:12:21.482764 2544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:21.482748 2546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:21.483091 2538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:21.483305 2538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:21.484468 2538 hybrid_clock.cc:648] HybridClock initialized: now 1769829141484445 us; error 32 us; skew 500 ppm
Jan 31 03:12:21 dist-test-slave-5nnp krb5kdc[1223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829141, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:21.486979 2538 init.cc:377] Logged in from keytab as kudu/127.19.84.2@KRBTEST.COM (short username kudu)
I20260131 03:12:21.487679 2538 webserver.cc:492] Webserver started at http://127.19.84.2:34399/ using document root <none> and password file <none>
I20260131 03:12:21.487891 2538 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:21.487950 2538 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:21.489193 2538 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:12:21.489768 2553 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:21.489984 2538 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260131 03:12:21.490072 2538 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "f9f2dc8e777c45fc971651de8f01f690"
format_stamp: "Formatted at 2026-01-31 03:12:20 on dist-test-slave-5nnp"
server_key: "c260a49d134b2599aee18aacbcca8528"
server_key_iv: "95cfd6c7349fb3d6780b261f08408e1a"
server_key_version: "encryptionkey@0"
I20260131 03:12:21.490403 2538 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:21.506220 2538 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:21.508417 2538 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:21.508584 2538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:21.508824 2538 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:12:21.509146 2538 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:12:21.509214 2538 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:21.509279 2538 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:12:21.509310 2538 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:21.515807 2538 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:42987
I20260131 03:12:21.515857 2666 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:42987 every 8 connection(s)
I20260131 03:12:21.516299 2538 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
I20260131 03:12:21.517436 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:21.516989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.2:40897)
0131 03:12:21.517168 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:21.517264 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
W20260131 03:12:21.517680 2667 heartbeater.cc:646] Failed to heartbeat to 127.19.84.62:41917 (0 consecutive failures): Network error: Failed to ping master at 127.19.84.62:41917: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
I20260131 03:12:21.517967 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:21.517870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.2:44601)
0131 03:12:21.517919 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:21.517944 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260131 03:12:21.518268 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:21.518184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.2:53485)
0131 03:12:21.518228 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:21.518249 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260131 03:12:21.518371 2667 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260131 03:12:21.520094 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 2538
I20260131 03:12:21.520358 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:39851
--local_ip_for_outbound_sockets=127.19.84.3
--tserver_master_addrs=127.19.84.62:41917
--webserver_port=38147
--webserver_interface=127.19.84.3
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:12:21.599095 2670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:21.599306 2670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:21.599344 2670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:21.600798 2670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:21.600867 2670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:21.600911 2670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:12:21.602509 2670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:39851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=38147
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:41917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:21.602761 2670 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:21.602949 2670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:21.605661 2675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:21.605751 2676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:21.605667 2678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:21.605845 2670 server_base.cc:1060] running on GCE node
I20260131 03:12:21.606026 2670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:21.606209 2670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:21.607442 2670 hybrid_clock.cc:648] HybridClock initialized: now 1769829141607410 us; error 40 us; skew 500 ppm
Jan 31 03:12:21 dist-test-slave-5nnp krb5kdc[1223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829141, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:21.609805 2670 init.cc:377] Logged in from keytab as kudu/127.19.84.3@KRBTEST.COM (short username kudu)
I20260131 03:12:21.610330 2670 webserver.cc:492] Webserver started at http://127.19.84.3:38147/ using document root <none> and password file <none>
I20260131 03:12:21.610533 2670 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:21.610590 2670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:21.611884 2670 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:12:21.612442 2685 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:21.612615 2670 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260131 03:12:21.612700 2670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "2c2bad11ca33403eb47410763aa631e6"
format_stamp: "Formatted at 2026-01-31 03:12:20 on dist-test-slave-5nnp"
server_key: "355a3d381a466f666c5140d09ec65643"
server_key_iv: "2cca9d643fb5fdb2cd070cc33caff6e2"
server_key_version: "encryptionkey@0"
I20260131 03:12:21.612977 2670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:21.628108 2670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:21.629683 2670 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:21.629840 2670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:21.630062 2670 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:12:21.630365 2670 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:12:21.630419 2670 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:21.630473 2670 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:12:21.630497 2670 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:21.636220 2670 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:39851
I20260131 03:12:21.636266 2798 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:39851 every 8 connection(s)
I20260131 03:12:21.636633 2670 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
I20260131 03:12:21.637544 2801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:21.637236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.3:47635)
0131 03:12:21.637367 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:21.637434 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":67,"threads_started":1}
W20260131 03:12:21.637784 2799 heartbeater.cc:646] Failed to heartbeat to 127.19.84.62:41917 (0 consecutive failures): Network error: Failed to ping master at 127.19.84.62:41917: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
I20260131 03:12:21.638067 2801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:21.637984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.3:39877)
0131 03:12:21.638026 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:21.638050 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260131 03:12:21.638337 2801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:21.638264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.3:56421)
0131 03:12:21.638302 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:21.638323 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260131 03:12:21.638447 2799 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260131 03:12:21.644702 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 2670
I20260131 03:12:22.474578 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:22.474273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.1:46591)
0131 03:12:22.474469 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:22.474524 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":79,"threads_started":1}
I20260131 03:12:22.519068 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:22.518730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.2:59407)
0131 03:12:22.518908 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:22.518966 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":70,"threads_started":1}
I20260131 03:12:22.639102 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:22.638784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.3:43925)
0131 03:12:22.638978 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:22.639029 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
I20260131 03:12:23.060976 2806 raft_consensus.cc:493] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:12:23.061115 2806 raft_consensus.cc:515] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } } peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } }
I20260131 03:12:23.061400 2806 leader_election.cc:290] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9f2dc8e777c45fc971651de8f01f690 (127.19.84.2:42987), 2c2bad11ca33403eb47410763aa631e6 (127.19.84.3:39851)
Jan 31 03:12:23 dist-test-slave-5nnp krb5kdc[1223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829141, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.3@KRBTEST.COM
Jan 31 03:12:23 dist-test-slave-5nnp krb5kdc[1223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829141, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.2@KRBTEST.COM
I20260131 03:12:23.071025 2810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:23.061874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:56069 (local address 127.19.84.3:39851)
0131 03:12:23.061983 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:23.061997 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:23.062011 (+ 14us) server_negotiation.cc:408] Connection header received
0131 03:12:23.062433 (+ 422us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:23.062436 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:23.062469 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:23.062609 (+ 140us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:23.063555 (+ 946us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:23.064641 (+ 1086us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:23.065271 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:23.065549 (+ 278us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:23.067557 (+ 2008us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:23.067565 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:23.067565 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:23.067604 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:23.069519 (+ 1915us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:23.069996 (+ 477us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:23.069999 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:23.069999 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:23.070057 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:23.070257 (+ 200us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:23.070258 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:23.070258 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:23.070453 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:23.070559 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:23.070813 (+ 254us) server_negotiation.cc:300] Negotiation successful
0131 03:12:23.070922 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":54,"threads_started":1}
I20260131 03:12:23.071017 2809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:23.061760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:39851 (local address 127.19.84.1:56069)
0131 03:12:23.061874 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:23.061883 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:23.061936 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:23.062618 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:23.062619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:23.063024 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:23.063420 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:23.063425 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:23.064741 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:23.064743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:23.065188 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:23.065190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:23.065275 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:23.065857 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:23.065884 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:23.067445 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:23.069644 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:23.069649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:23.069652 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:23.069884 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:23.070126 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:23.070126 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:23.070127 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:23.070192 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:23.070528 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:23.070533 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:23.070652 (+ 119us) client_negotiation.cc:769] Sending connection context
0131 03:12:23.070762 (+ 110us) client_negotiation.cc:240] Negotiation successful
0131 03:12:23.070911 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":59,"threads_started":1}
I20260131 03:12:23.071089 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:23.061709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:41807 (local address 127.19.84.2:42987)
0131 03:12:23.061847 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:23.061857 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:23.061882 (+ 25us) server_negotiation.cc:408] Connection header received
0131 03:12:23.062383 (+ 501us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:23.062386 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:23.062426 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:23.062525 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:23.063551 (+ 1026us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:23.064564 (+ 1013us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:23.065133 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:23.065379 (+ 246us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:23.067928 (+ 2549us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:23.067935 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:23.067935 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:23.067954 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:23.069631 (+ 1677us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:23.070014 (+ 383us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:23.070017 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:23.070017 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:23.070167 (+ 150us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:23.070379 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:23.070380 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:23.070380 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:23.070566 (+ 186us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:23.070669 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:23.070844 (+ 175us) server_negotiation.cc:300] Negotiation successful
0131 03:12:23.070978 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":56,"threads_started":1}
I20260131 03:12:23.071135 2807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:23.061594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:42987 (local address 127.19.84.1:41807)
0131 03:12:23.061705 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:23.061715 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:23.061813 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:23.062594 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:23.062597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:23.063010 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:23.063420 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:23.063425 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:23.064679 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:23.064680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:23.065051 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:23.065052 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:23.065194 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:23.065953 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:23.065957 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:23.067815 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:23.069732 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:23.069734 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:23.069734 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:23.069920 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:23.070275 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:23.070276 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:23.070276 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:23.070313 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:23.070643 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:23.070644 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:23.070686 (+ 42us) client_negotiation.cc:769] Sending connection context
0131 03:12:23.070780 (+ 94us) client_negotiation.cc:240] Negotiation successful
0131 03:12:23.071005 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":111,"thread_start_us":61,"threads_started":1}
W20260131 03:12:23.072067 2419 leader_election.cc:343] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9f2dc8e777c45fc971651de8f01f690 (127.19.84.2:42987): Not found: Tablet not found: 4748055821b043e49b2a947a6718824f
W20260131 03:12:23.072212 2419 leader_election.cc:343] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2c2bad11ca33403eb47410763aa631e6 (127.19.84.3:39851): Not found: Tablet not found: 4748055821b043e49b2a947a6718824f
I20260131 03:12:23.072257 2419 leader_election.cc:304] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90c0155151a74528816bcf8a9287396d; no voters: 2c2bad11ca33403eb47410763aa631e6, f9f2dc8e777c45fc971651de8f01f690
I20260131 03:12:23.072347 2806 raft_consensus.cc:2749] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260131 03:12:23.475348 2809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:23.475217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.1:45603)
0131 03:12:23.475270 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:23.475300 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260131 03:12:23.519819 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:23.519583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.2:42339)
0131 03:12:23.519726 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:23.519775 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":59,"threads_started":1}
I20260131 03:12:23.639791 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:23.639603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.3:36105)
0131 03:12:23.639717 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:23.639751 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
I20260131 03:12:24.476236 2813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:24.475939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.1:56381)
0131 03:12:24.476095 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:24.476143 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260131 03:12:24.520571 2814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:24.520352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.2:52311)
0131 03:12:24.520486 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:24.520528 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260131 03:12:24.640481 2815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:24.640242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.3:55779)
0131 03:12:24.640386 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:24.640424 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":57,"threads_started":1}
I20260131 03:12:25.203240 2816 raft_consensus.cc:493] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:12:25.203378 2816 raft_consensus.cc:515] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } } peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } }
I20260131 03:12:25.203629 2816 leader_election.cc:290] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9f2dc8e777c45fc971651de8f01f690 (127.19.84.2:42987), 2c2bad11ca33403eb47410763aa631e6 (127.19.84.3:39851)
W20260131 03:12:25.204066 2419 leader_election.cc:343] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2c2bad11ca33403eb47410763aa631e6 (127.19.84.3:39851): Not found: Tablet not found: 4748055821b043e49b2a947a6718824f
W20260131 03:12:25.204149 2419 leader_election.cc:343] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9f2dc8e777c45fc971651de8f01f690 (127.19.84.2:42987): Not found: Tablet not found: 4748055821b043e49b2a947a6718824f
I20260131 03:12:25.204164 2419 leader_election.cc:304] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90c0155151a74528816bcf8a9287396d; no voters: 2c2bad11ca33403eb47410763aa631e6, f9f2dc8e777c45fc971651de8f01f690
I20260131 03:12:25.204247 2816 raft_consensus.cc:2749] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260131 03:12:25.476953 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:25.476742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.1:53249)
0131 03:12:25.476862 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:25.476902 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260131 03:12:25.521318 2818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:25.521065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.2:35503)
0131 03:12:25.521219 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:25.521259 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
I20260131 03:12:25.641119 2819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:25.640898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.3:41301)
0131 03:12:25.641039 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:25.641076 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
I20260131 03:12:26.477705 2820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:26.477475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.1:43275)
0131 03:12:26.477613 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:26.477649 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260131 03:12:26.522027 2821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:26.521797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.2:35333)
0131 03:12:26.521943 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:26.521982 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260131 03:12:26.641862 2822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:26.641627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.3:58707)
0131 03:12:26.641771 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:26.641814 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260131 03:12:27.478616 2823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:27.478375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.1:48533)
0131 03:12:27.478523 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:27.478566 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
I20260131 03:12:27.522873 2824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:27.522632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.2:57189)
0131 03:12:27.522785 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:27.522826 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":65,"threads_started":1}
I20260131 03:12:27.642597 2825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:27.642353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.3:33935)
0131 03:12:27.642518 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:27.642551 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":92,"threads_started":1}
I20260131 03:12:27.740432 2826 raft_consensus.cc:493] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:12:27.740564 2826 raft_consensus.cc:515] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } } peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } }
I20260131 03:12:27.740757 2826 leader_election.cc:290] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9f2dc8e777c45fc971651de8f01f690 (127.19.84.2:42987), 2c2bad11ca33403eb47410763aa631e6 (127.19.84.3:39851)
W20260131 03:12:27.741166 2419 leader_election.cc:343] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9f2dc8e777c45fc971651de8f01f690 (127.19.84.2:42987): Not found: Tablet not found: 4748055821b043e49b2a947a6718824f
W20260131 03:12:27.741250 2419 leader_election.cc:343] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2c2bad11ca33403eb47410763aa631e6 (127.19.84.3:39851): Not found: Tablet not found: 4748055821b043e49b2a947a6718824f
I20260131 03:12:27.741269 2419 leader_election.cc:304] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90c0155151a74528816bcf8a9287396d; no voters: 2c2bad11ca33403eb47410763aa631e6, f9f2dc8e777c45fc971651de8f01f690
I20260131 03:12:27.741340 2826 raft_consensus.cc:2749] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260131 03:12:28.479456 2827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:28.479188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.1:57161)
0131 03:12:28.479318 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:28.479355 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260131 03:12:28.523545 2828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:28.523302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.2:55839)
0131 03:12:28.523463 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:28.523499 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":60,"threads_started":1}
I20260131 03:12:28.643424 2829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:28.643158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.3:33285)
0131 03:12:28.643296 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:28.643332 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
I20260131 03:12:29.480221 2830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:29.479971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.1:56253)
0131 03:12:29.480121 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:29.480168 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":69,"threads_started":1}
I20260131 03:12:29.524238 2831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:29.524010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.2:48703)
0131 03:12:29.524129 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:29.524168 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260131 03:12:29.644178 2832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:29.643975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.3:53677)
0131 03:12:29.644095 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:29.644132 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260131 03:12:30.480955 2833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:30.480705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.1:46277)
0131 03:12:30.480860 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:30.480903 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":63,"threads_started":1}
I20260131 03:12:30.524982 2834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:30.524747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.2:57575)
0131 03:12:30.524897 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:30.524934 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":74,"threads_started":1}
I20260131 03:12:30.644938 2835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:30.644706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.3:54341)
0131 03:12:30.644846 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:30.644890 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
I20260131 03:12:31.272876 2836 raft_consensus.cc:493] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:12:31.272980 2836 raft_consensus.cc:515] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } } peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } }
I20260131 03:12:31.273169 2836 leader_election.cc:290] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9f2dc8e777c45fc971651de8f01f690 (127.19.84.2:42987), 2c2bad11ca33403eb47410763aa631e6 (127.19.84.3:39851)
W20260131 03:12:31.273666 2419 leader_election.cc:343] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2c2bad11ca33403eb47410763aa631e6 (127.19.84.3:39851): Not found: Tablet not found: 4748055821b043e49b2a947a6718824f
W20260131 03:12:31.273744 2419 leader_election.cc:343] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9f2dc8e777c45fc971651de8f01f690 (127.19.84.2:42987): Not found: Tablet not found: 4748055821b043e49b2a947a6718824f
I20260131 03:12:31.273761 2419 leader_election.cc:304] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90c0155151a74528816bcf8a9287396d; no voters: 2c2bad11ca33403eb47410763aa631e6, f9f2dc8e777c45fc971651de8f01f690
I20260131 03:12:31.273883 2836 raft_consensus.cc:2749] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260131 03:12:31.481763 2837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:31.481505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.1:40611)
0131 03:12:31.481669 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:31.481711 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
I20260131 03:12:31.525919 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:31.525663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.2:59125)
0131 03:12:31.525824 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:31.525868 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":72,"threads_started":1}
I20260131 03:12:31.645682 2839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:31.645461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.3:43857)
0131 03:12:31.645596 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:31.645636 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260131 03:12:32.482640 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:32.482360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.1:38565)
0131 03:12:32.482553 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:32.482590 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":130,"threads_started":1}
I20260131 03:12:32.526826 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:32.526545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.2:35195)
0131 03:12:32.526714 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:32.526747 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":113,"threads_started":1}
I20260131 03:12:32.646374 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:32.646150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.3:48045)
0131 03:12:32.646274 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:32.646310 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
I20260131 03:12:33.483459 2843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:33.483157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.1:37925)
0131 03:12:33.483312 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:33.483355 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260131 03:12:33.527607 2844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:33.527320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.2:36609)
0131 03:12:33.527513 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:33.527558 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":135,"threads_started":1}
I20260131 03:12:33.647037 2845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:33.646816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.3:49331)
0131 03:12:33.646956 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:33.646989 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260131 03:12:34.484227 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:34.484010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.1:38085)
0131 03:12:34.484138 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:34.484177 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":65,"threads_started":1}
I20260131 03:12:34.528369 2847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:34.528120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.2:36017)
0131 03:12:34.528275 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:34.528316 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20260131 03:12:34.647819 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:34.647589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.3:60961)
0131 03:12:34.647730 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:34.647771 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260131 03:12:34.651894 2849 raft_consensus.cc:493] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:12:34.652014 2849 raft_consensus.cc:515] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9f2dc8e777c45fc971651de8f01f690" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42987 } } peers { permanent_uuid: "2c2bad11ca33403eb47410763aa631e6" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39851 } } peers { permanent_uuid: "90c0155151a74528816bcf8a9287396d" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35801 } }
I20260131 03:12:34.652197 2849 leader_election.cc:290] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9f2dc8e777c45fc971651de8f01f690 (127.19.84.2:42987), 2c2bad11ca33403eb47410763aa631e6 (127.19.84.3:39851)
W20260131 03:12:34.652630 2419 leader_election.cc:343] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9f2dc8e777c45fc971651de8f01f690 (127.19.84.2:42987): Not found: Tablet not found: 4748055821b043e49b2a947a6718824f
W20260131 03:12:34.652693 2419 leader_election.cc:343] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2c2bad11ca33403eb47410763aa631e6 (127.19.84.3:39851): Not found: Tablet not found: 4748055821b043e49b2a947a6718824f
I20260131 03:12:34.652709 2419 leader_election.cc:304] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90c0155151a74528816bcf8a9287396d; no voters: 2c2bad11ca33403eb47410763aa631e6, f9f2dc8e777c45fc971651de8f01f690
I20260131 03:12:34.652813 2849 raft_consensus.cc:2749] T 4748055821b043e49b2a947a6718824f P 90c0155151a74528816bcf8a9287396d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260131 03:12:35.485051 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:35.484805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.1:53491)
0131 03:12:35.484956 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:35.485001 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260131 03:12:35.529052 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:35.528829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.2:34793)
0131 03:12:35.528969 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:35.529003 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
I20260131 03:12:35.648640 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:35.648412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.3:53209)
0131 03:12:35.648553 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:35.648591 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260131 03:12:36.485889 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:36.485635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.1:36047)
0131 03:12:36.485791 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:36.485836 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":63,"threads_started":1}
I20260131 03:12:36.529943 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:36.529701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.2:50815)
0131 03:12:36.529851 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:36.529895 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":68,"threads_started":1}
I20260131 03:12:36.649590 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:36.649293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41917 (local address 127.19.84.3:46927)
0131 03:12:36.649485 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:36.649531 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":82,"threads_started":1}
I20260131 03:12:36.651044 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 2374
I20260131 03:12:36.656168 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 2538
I20260131 03:12:36.660866 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 2670
2026-01-31T03:12:36Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19201 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:12:36 dist-test-slave-5nnp krb5kdc[2858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:12:36 dist-test-slave-5nnp krb5kdc[2858](info): set up 2 sockets
Jan 31 03:12:36 dist-test-slave-5nnp krb5kdc[2858](info): commencing operation
krb5kdc: starting...
W20260131 03:12:38.668406 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.987s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:12:38 dist-test-slave-5nnp krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:12:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:12:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:12:38.810299 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:44007
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:44007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:12:38.889149 2874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:38.889408 2874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:38.889451 2874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:38.890933 2874 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260131 03:12:38.891010 2874 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260131 03:12:38.891031 2874 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260131 03:12:38.891049 2874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:12:38.891078 2874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:38.891108 2874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:38.891134 2874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:12:38.891162 2874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:12:38.892860 2874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:44007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:44007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:38.893126 2874 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:38.893342 2874 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:38.895888 2882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:38.895882 2880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:38.895939 2874 server_base.cc:1060] running on GCE node
W20260131 03:12:38.895882 2879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:38.896224 2874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:38.896490 2874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:38.897662 2874 hybrid_clock.cc:648] HybridClock initialized: now 1769829158897645 us; error 31 us; skew 500 ppm
Jan 31 03:12:38 dist-test-slave-5nnp krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829158, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:38.900525 2874 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:12:38.901185 2874 webserver.cc:492] Webserver started at http://127.19.84.62:45217/ using document root <none> and password file <none>
I20260131 03:12:38.901387 2874 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:38.901449 2874 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:38.901594 2874 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:12:38.902582 2874 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "ac601929b712426b9fdd57c25bb41aff"
format_stamp: "Formatted at 2026-01-31 03:12:38 on dist-test-slave-5nnp"
server_key: "acb4290367c4772ed1cf87036370060f"
server_key_iv: "a4724bbcd22ec22e928c6ba195f0c29b"
server_key_version: "encryptionkey@0"
I20260131 03:12:38.902904 2874 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "ac601929b712426b9fdd57c25bb41aff"
format_stamp: "Formatted at 2026-01-31 03:12:38 on dist-test-slave-5nnp"
server_key: "acb4290367c4772ed1cf87036370060f"
server_key_iv: "a4724bbcd22ec22e928c6ba195f0c29b"
server_key_version: "encryptionkey@0"
I20260131 03:12:38.904316 2874 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:12:38.905186 2889 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:38.905421 2874 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:12:38.905524 2874 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "ac601929b712426b9fdd57c25bb41aff"
format_stamp: "Formatted at 2026-01-31 03:12:38 on dist-test-slave-5nnp"
server_key: "acb4290367c4772ed1cf87036370060f"
server_key_iv: "a4724bbcd22ec22e928c6ba195f0c29b"
server_key_version: "encryptionkey@0"
I20260131 03:12:38.905609 2874 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:38.931691 2874 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:38.933843 2874 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:38.934005 2874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:38.938462 2874 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:44007
I20260131 03:12:38.938501 2941 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:44007 every 8 connection(s)
I20260131 03:12:38.938823 2874 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:12:38.939600 2942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:12:38.942574 2942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff: Bootstrap starting.
I20260131 03:12:38.943298 2942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff: Neither blocks nor log segments found. Creating new log.
I20260131 03:12:38.943646 2942 log.cc:826] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:38.944428 2942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff: No bootstrap required, opened a new log
I20260131 03:12:38.945355 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 2874
I20260131 03:12:38.945472 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:12:38.945636 19792 external_mini_cluster.cc:1468] Setting key 869e03294dee5d04fbe5ad29495a2c25
I20260131 03:12:38.946112 2942 raft_consensus.cc:359] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac601929b712426b9fdd57c25bb41aff" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44007 } }
I20260131 03:12:38.946272 2942 raft_consensus.cc:385] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:12:38.946310 2942 raft_consensus.cc:740] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac601929b712426b9fdd57c25bb41aff, State: Initialized, Role: FOLLOWER
I20260131 03:12:38.946458 2942 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac601929b712426b9fdd57c25bb41aff" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44007 } }
I20260131 03:12:38.946528 2942 raft_consensus.cc:399] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:12:38.946559 2942 raft_consensus.cc:493] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:12:38.946607 2942 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:12:38.947183 2942 raft_consensus.cc:515] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac601929b712426b9fdd57c25bb41aff" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44007 } }
I20260131 03:12:38.947309 2942 leader_election.cc:304] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac601929b712426b9fdd57c25bb41aff; no voters:
I20260131 03:12:38.947495 2942 leader_election.cc:290] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:12:38.947557 2947 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:12:38.947737 2947 raft_consensus.cc:697] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 1 LEADER]: Becoming Leader. State: Replica: ac601929b712426b9fdd57c25bb41aff, State: Running, Role: LEADER
I20260131 03:12:38.947762 2942 sys_catalog.cc:565] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:12:38.947878 2947 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac601929b712426b9fdd57c25bb41aff" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44007 } }
I20260131 03:12:38.948191 2947 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac601929b712426b9fdd57c25bb41aff. Latest consensus state: current_term: 1 leader_uuid: "ac601929b712426b9fdd57c25bb41aff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac601929b712426b9fdd57c25bb41aff" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44007 } } }
I20260131 03:12:38.948199 2948 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac601929b712426b9fdd57c25bb41aff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac601929b712426b9fdd57c25bb41aff" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44007 } } }
I20260131 03:12:38.948418 2948 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [sys.catalog]: This master's current role is: LEADER
I20260131 03:12:38.948612 2947 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [sys.catalog]: This master's current role is: LEADER
I20260131 03:12:38.949298 2955 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:12:38.950541 2955 catalog_manager.cc:1494] Initializing Kudu cluster ID...
Jan 31 03:12:38 dist-test-slave-5nnp krb5kdc[2858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:12:38.952214 2955 catalog_manager.cc:1357] Generated new cluster ID: 3d3ff55d824846cab7c1c66c4ae6e4b8
I20260131 03:12:38.952271 2955 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:12:38.956507 2946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:38.946168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34288 (local address 127.19.84.62:44007)
0131 03:12:38.946368 (+ 200us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:38.946380 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:38.946405 (+ 25us) server_negotiation.cc:408] Connection header received
0131 03:12:38.946513 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:38.946520 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:38.946572 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:38.946709 (+ 137us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:38.947646 (+ 937us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:38.948908 (+ 1262us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:38.949739 (+ 831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:38.949912 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:38.952721 (+ 2809us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:38.952743 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:38.952743 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:38.952764 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:38.954847 (+ 2083us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:38.955242 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:38.955245 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:38.955245 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:38.955322 (+ 77us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:38.955526 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:38.955528 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:38.955528 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:38.955793 (+ 265us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:38.955920 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:38.956098 (+ 178us) server_negotiation.cc:300] Negotiation successful
0131 03:12:38.956332 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":88,"threads_started":1}
I20260131 03:12:38.959946 2955 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:12:38.960485 2955 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:12:38.968950 2955 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff: Generated new TSK 0
I20260131 03:12:38.969192 2955 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:12:39.030854 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:44007
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:12:39.110747 2970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:39.110934 2970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:39.110960 2970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:39.112430 2970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:39.112478 2970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:39.112540 2970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:12:39.114131 2970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:44007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:39.114364 2970 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:39.114566 2970 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:39.117177 2978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:39.117170 2976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:39.117432 2970 server_base.cc:1060] running on GCE node
W20260131 03:12:39.117170 2975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:39.117651 2970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:39.117822 2970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:39.118959 2970 hybrid_clock.cc:648] HybridClock initialized: now 1769829159118939 us; error 33 us; skew 500 ppm
Jan 31 03:12:39 dist-test-slave-5nnp krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829159, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:39.121333 2970 init.cc:377] Logged in from keytab as kudu/127.19.84.1@KRBTEST.COM (short username kudu)
I20260131 03:12:39.121863 2970 webserver.cc:492] Webserver started at http://127.19.84.1:36309/ using document root <none> and password file <none>
I20260131 03:12:39.122061 2970 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:39.122099 2970 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:39.122231 2970 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:12:39.123157 2970 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "0e89581941ff47089b5f0a801f9316a0"
format_stamp: "Formatted at 2026-01-31 03:12:39 on dist-test-slave-5nnp"
server_key: "c2752560b142f91111774841f87be240"
server_key_iv: "f3582a8173abc62bd818f023a3e49f13"
server_key_version: "encryptionkey@0"
I20260131 03:12:39.123510 2970 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "0e89581941ff47089b5f0a801f9316a0"
format_stamp: "Formatted at 2026-01-31 03:12:39 on dist-test-slave-5nnp"
server_key: "c2752560b142f91111774841f87be240"
server_key_iv: "f3582a8173abc62bd818f023a3e49f13"
server_key_version: "encryptionkey@0"
I20260131 03:12:39.124692 2970 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:12:39.125409 2985 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:39.125592 2970 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:39.125648 2970 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "0e89581941ff47089b5f0a801f9316a0"
format_stamp: "Formatted at 2026-01-31 03:12:39 on dist-test-slave-5nnp"
server_key: "c2752560b142f91111774841f87be240"
server_key_iv: "f3582a8173abc62bd818f023a3e49f13"
server_key_version: "encryptionkey@0"
I20260131 03:12:39.125727 2970 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:39.149827 2970 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:39.151859 2970 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:39.152014 2970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:39.152240 2970 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:12:39.152552 2970 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:12:39.152585 2970 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:39.152609 2970 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:12:39.152654 2970 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:39.158066 2970 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:43179
I20260131 03:12:39.158187 3098 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:43179 every 8 connection(s)
I20260131 03:12:39.158440 2970 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
Jan 31 03:12:39 dist-test-slave-5nnp krb5kdc[2858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829159, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:12:39.165563 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 2970
I20260131 03:12:39.165650 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:12:39.165840 19792 external_mini_cluster.cc:1468] Setting key e85f0f4a9b68d33b3b5d626bd251c86a
I20260131 03:12:39.166740 2946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:39.159124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:33915 (local address 127.19.84.62:44007)
0131 03:12:39.159174 (+ 50us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:39.159177 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:39.159358 (+ 181us) server_negotiation.cc:408] Connection header received
0131 03:12:39.159923 (+ 565us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:39.159925 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:39.159940 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:39.159999 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:39.160755 (+ 756us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.161274 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:39.161828 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.161945 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:39.164071 (+ 2126us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:39.164074 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:39.164075 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:39.164080 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:39.165302 (+ 1222us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:39.165737 (+ 435us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:39.165739 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:39.165740 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:39.165769 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:39.166023 (+ 254us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:39.166024 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:39.166025 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:39.166164 (+ 139us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:39.166242 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:39.166579 (+ 337us) server_negotiation.cc:300] Negotiation successful
0131 03:12:39.166675 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260131 03:12:39.166960 3101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:39.159089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.1:33915)
0131 03:12:39.159261 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:39.159289 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:39.159808 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:39.160066 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:39.160070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:39.160384 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:39.160652 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:39.160654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.161353 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:39.161354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:39.161751 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:39.161752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.161855 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:39.162638 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:39.162657 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:39.163995 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:39.165385 (+ 1390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:39.165391 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:39.165393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:39.165594 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:39.165839 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:39.165840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:39.165840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:39.165930 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:39.166241 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:39.166246 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:39.166393 (+ 147us) client_negotiation.cc:769] Sending connection context
0131 03:12:39.166585 (+ 192us) client_negotiation.cc:240] Negotiation successful
0131 03:12:39.166735 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260131 03:12:39.167513 3099 heartbeater.cc:344] Connected to a master server at 127.19.84.62:44007
I20260131 03:12:39.167622 3099 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:39.167821 3099 heartbeater.cc:507] Master 127.19.84.62:44007 requested a full tablet report, sending...
I20260131 03:12:39.168376 2906 ts_manager.cc:194] Registered new tserver with Master: 0e89581941ff47089b5f0a801f9316a0 (127.19.84.1:43179)
I20260131 03:12:39.169004 2906 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.1@KRBTEST.COM'} at 127.19.84.1:33915
WARNING: no policy specified for kudu/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:12:39.213655 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:44007
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:12:39.291623 3106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:39.291795 3106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:39.291817 3106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:39.293270 3106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:39.293313 3106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:39.293339 3106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:12:39.294989 3106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:44007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:39.295217 3106 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:39.295461 3106 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:39.297932 3112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:39.297926 3114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:39.297926 3111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:39.298080 3106 server_base.cc:1060] running on GCE node
I20260131 03:12:39.298311 3106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:39.298486 3106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:39.299621 3106 hybrid_clock.cc:648] HybridClock initialized: now 1769829159299602 us; error 37 us; skew 500 ppm
Jan 31 03:12:39 dist-test-slave-5nnp krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829159, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:39.301867 3106 init.cc:377] Logged in from keytab as kudu/127.19.84.2@KRBTEST.COM (short username kudu)
I20260131 03:12:39.302326 3106 webserver.cc:492] Webserver started at http://127.19.84.2:35963/ using document root <none> and password file <none>
I20260131 03:12:39.302476 3106 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:39.302513 3106 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:39.302589 3106 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:12:39.303546 3106 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "7a22130f7cb84aa1a2cf411ddf784ec9"
format_stamp: "Formatted at 2026-01-31 03:12:39 on dist-test-slave-5nnp"
server_key: "f13125ac5206639bba894a27e60e0b1a"
server_key_iv: "58048bfdca9c92acdd187bda64e024bd"
server_key_version: "encryptionkey@0"
I20260131 03:12:39.303812 3106 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "7a22130f7cb84aa1a2cf411ddf784ec9"
format_stamp: "Formatted at 2026-01-31 03:12:39 on dist-test-slave-5nnp"
server_key: "f13125ac5206639bba894a27e60e0b1a"
server_key_iv: "58048bfdca9c92acdd187bda64e024bd"
server_key_version: "encryptionkey@0"
I20260131 03:12:39.304904 3106 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:12:39.305502 3121 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:39.305692 3106 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260131 03:12:39.305759 3106 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "7a22130f7cb84aa1a2cf411ddf784ec9"
format_stamp: "Formatted at 2026-01-31 03:12:39 on dist-test-slave-5nnp"
server_key: "f13125ac5206639bba894a27e60e0b1a"
server_key_iv: "58048bfdca9c92acdd187bda64e024bd"
server_key_version: "encryptionkey@0"
I20260131 03:12:39.305805 3106 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:39.317350 3106 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:39.319568 3106 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:39.319674 3106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:39.319851 3106 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:12:39.320101 3106 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:12:39.320129 3106 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:39.320151 3106 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:12:39.320164 3106 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:39.326082 3106 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:46289
I20260131 03:12:39.326174 3234 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:46289 every 8 connection(s)
I20260131 03:12:39.326486 3106 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
I20260131 03:12:39.327648 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 3106
I20260131 03:12:39.327754 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:12:39.327931 19792 external_mini_cluster.cc:1468] Setting key db1b0f86782c49b190a3600dcc242130
Jan 31 03:12:39 dist-test-slave-5nnp krb5kdc[2858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829159, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.84.3@KRBTEST.COM; defaulting to no policy
I20260131 03:12:39.336206 2946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:39.327262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:40691 (local address 127.19.84.62:44007)
0131 03:12:39.327361 (+ 99us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:39.327363 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:39.327524 (+ 161us) server_negotiation.cc:408] Connection header received
0131 03:12:39.328184 (+ 660us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:39.328186 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:39.328198 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:39.328273 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:39.329347 (+ 1074us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.329839 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:39.330498 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.330711 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:39.333426 (+ 2715us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:39.333430 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:39.333431 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:39.333439 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:39.334844 (+ 1405us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:39.335254 (+ 410us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:39.335255 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:39.335256 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:39.335283 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:39.335536 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:39.335537 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:39.335538 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:39.335641 (+ 103us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:39.335721 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:39.336009 (+ 288us) server_negotiation.cc:300] Negotiation successful
0131 03:12:39.336109 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260131 03:12:39.336328 3237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:39.327202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.2:40691)
0131 03:12:39.327347 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:39.327414 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:39.328072 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:39.328341 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:39.328345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:39.328804 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:39.329231 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:39.329236 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.329929 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:39.329930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:39.330409 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:39.330410 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.330560 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:39.331635 (+ 1075us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:39.331660 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:39.333316 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:39.334955 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:39.334962 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:39.334965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:39.335181 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:39.335358 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:39.335359 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:39.335360 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:39.335464 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:39.335708 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:39.335714 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:39.335871 (+ 157us) client_negotiation.cc:769] Sending connection context
0131 03:12:39.335979 (+ 108us) client_negotiation.cc:240] Negotiation successful
0131 03:12:39.336114 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260131 03:12:39.336777 3235 heartbeater.cc:344] Connected to a master server at 127.19.84.62:44007
I20260131 03:12:39.336874 3235 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:39.337065 3235 heartbeater.cc:507] Master 127.19.84.62:44007 requested a full tablet report, sending...
I20260131 03:12:39.337421 2906 ts_manager.cc:194] Registered new tserver with Master: 7a22130f7cb84aa1a2cf411ddf784ec9 (127.19.84.2:46289)
I20260131 03:12:39.337801 2906 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.2@KRBTEST.COM'} at 127.19.84.2:40691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:12:39.373948 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:44007
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:12:39.452044 3242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:39.452245 3242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:39.452289 3242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:39.453723 3242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:39.453796 3242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:39.453840 3242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:12:39.455540 3242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:44007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:39.455787 3242 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:39.455983 3242 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:39.458545 3247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:39.458556 3248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:39.458601 3242 server_base.cc:1060] running on GCE node
W20260131 03:12:39.458545 3250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:39.458959 3242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:39.459131 3242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:39.460261 3242 hybrid_clock.cc:648] HybridClock initialized: now 1769829159460250 us; error 27 us; skew 500 ppm
Jan 31 03:12:39 dist-test-slave-5nnp krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829159, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:39.462615 3242 init.cc:377] Logged in from keytab as kudu/127.19.84.3@KRBTEST.COM (short username kudu)
I20260131 03:12:39.463137 3242 webserver.cc:492] Webserver started at http://127.19.84.3:40797/ using document root <none> and password file <none>
I20260131 03:12:39.463357 3242 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:39.463447 3242 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:39.463564 3242 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:12:39.464504 3242 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "0b22a6e7c7344c2db0d2824179e31a35"
format_stamp: "Formatted at 2026-01-31 03:12:39 on dist-test-slave-5nnp"
server_key: "bb2f7fd82f3e459e95a3466a3241e01b"
server_key_iv: "d7f8bf897298dde5786b318310fe9221"
server_key_version: "encryptionkey@0"
I20260131 03:12:39.464814 3242 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "0b22a6e7c7344c2db0d2824179e31a35"
format_stamp: "Formatted at 2026-01-31 03:12:39 on dist-test-slave-5nnp"
server_key: "bb2f7fd82f3e459e95a3466a3241e01b"
server_key_iv: "d7f8bf897298dde5786b318310fe9221"
server_key_version: "encryptionkey@0"
I20260131 03:12:39.465981 3242 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:12:39.466681 3257 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:39.466849 3242 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260131 03:12:39.466933 3242 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "0b22a6e7c7344c2db0d2824179e31a35"
format_stamp: "Formatted at 2026-01-31 03:12:39 on dist-test-slave-5nnp"
server_key: "bb2f7fd82f3e459e95a3466a3241e01b"
server_key_iv: "d7f8bf897298dde5786b318310fe9221"
server_key_version: "encryptionkey@0"
I20260131 03:12:39.467005 3242 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:39.484172 3242 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:39.485930 3242 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:39.486083 3242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:39.486332 3242 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:12:39.486655 3242 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:12:39.486709 3242 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:39.486842 3242 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:12:39.486899 3242 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:39.493438 3242 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:37531
I20260131 03:12:39.493521 3370 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:37531 every 8 connection(s)
I20260131 03:12:39.493832 3242 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
I20260131 03:12:39.498009 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 3242
I20260131 03:12:39.498092 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:12:39.498281 19792 external_mini_cluster.cc:1468] Setting key 910555f205146fb4bf896c40186bca31
Jan 31 03:12:39 dist-test-slave-5nnp krb5kdc[2858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829159, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:12:39.502861 2946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:39.494576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:46775 (local address 127.19.84.62:44007)
0131 03:12:39.494663 (+ 87us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:39.494666 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:39.494831 (+ 165us) server_negotiation.cc:408] Connection header received
0131 03:12:39.495400 (+ 569us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:39.495402 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:39.495415 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:39.495470 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:39.496252 (+ 782us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.496755 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:39.497317 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.497488 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:39.500164 (+ 2676us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:39.500168 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:39.500168 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:39.500176 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:39.501588 (+ 1412us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:39.501999 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:39.502000 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:39.502000 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:39.502032 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:39.502229 (+ 197us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:39.502230 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:39.502230 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:39.502334 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:39.502410 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:39.502673 (+ 263us) server_negotiation.cc:300] Negotiation successful
0131 03:12:39.502769 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260131 03:12:39.502993 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:39.494526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.3:46775)
0131 03:12:39.494696 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:39.494732 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:39.495269 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:39.495539 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:39.495543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:39.495856 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:39.496125 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:39.496128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.496846 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:39.496847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:39.497231 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:39.497233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.497382 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:39.498444 (+ 1062us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:39.498475 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:39.500074 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:39.501694 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:39.501701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:39.501704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:39.501913 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:39.502094 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:39.502095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:39.502095 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:39.502161 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:39.502399 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:39.502404 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:39.502539 (+ 135us) client_negotiation.cc:769] Sending connection context
0131 03:12:39.502654 (+ 115us) client_negotiation.cc:240] Negotiation successful
0131 03:12:39.502788 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260131 03:12:39.503609 3371 heartbeater.cc:344] Connected to a master server at 127.19.84.62:44007
I20260131 03:12:39.503718 3371 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:39.503927 3371 heartbeater.cc:507] Master 127.19.84.62:44007 requested a full tablet report, sending...
I20260131 03:12:39.504346 2906 ts_manager.cc:194] Registered new tserver with Master: 0b22a6e7c7344c2db0d2824179e31a35 (127.19.84.3:37531)
I20260131 03:12:39.504753 2906 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:46775
I20260131 03:12:39.510250 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:12:39.517284 2946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:39.511637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34300 (local address 127.19.84.62:44007)
0131 03:12:39.511711 (+ 74us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:39.511713 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:39.511780 (+ 67us) server_negotiation.cc:408] Connection header received
0131 03:12:39.511836 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:39.511837 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:39.511846 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:39.511886 (+ 40us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:39.512593 (+ 707us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.513081 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:39.513702 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.513837 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:39.514822 (+ 985us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:39.514826 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:39.514826 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:39.514833 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:39.516272 (+ 1439us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:39.516552 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:39.516554 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:39.516554 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:39.516574 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:39.516756 (+ 182us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:39.516756 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:39.516757 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:39.516867 (+ 110us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:39.516946 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:39.517063 (+ 117us) server_negotiation.cc:300] Negotiation successful
0131 03:12:39.517131 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260131 03:12:39.518857 2906 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:12:39.520249 2906 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:12:39.526122 3383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:39.522841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:37531 (local address 127.0.0.1:48442)
0131 03:12:39.523011 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:39.523021 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:39.523078 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:39.523399 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:39.523402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:39.523403 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:39.523688 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:39.523690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.525063 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:39.525064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:39.525760 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:39.525762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.525825 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:39.525866 (+ 41us) client_negotiation.cc:769] Sending connection context
0131 03:12:39.525939 (+ 73us) client_negotiation.cc:240] Negotiation successful
0131 03:12:39.525988 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":114,"threads_started":1}
I20260131 03:12:39.526383 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:39.522799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:46289 (local address 127.0.0.1:45938)
0131 03:12:39.522947 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:39.522980 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:39.523054 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:39.523545 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:39.523545 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:39.523547 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:39.523728 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:39.523729 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.525020 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:39.525021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:39.525893 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:39.525895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.525957 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:39.525961 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:12:39.525997 (+ 36us) client_negotiation.cc:240] Negotiation successful
0131 03:12:39.526037 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260131 03:12:39.526543 3385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:39.523055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:43179 (local address 127.0.0.1:48342)
0131 03:12:39.523182 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:39.523192 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:39.523274 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:39.523877 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:39.523878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:39.523880 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:39.524064 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:39.524066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.524950 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:39.524951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:39.525880 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:39.525882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.525968 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:39.525972 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:12:39.526007 (+ 35us) client_negotiation.cc:240] Negotiation successful
0131 03:12:39.526037 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260131 03:12:39.526763 3387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:39.523461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48342 (local address 127.19.84.1:43179)
0131 03:12:39.523662 (+ 201us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:39.523670 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:39.523679 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:12:39.523716 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:39.523718 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:39.523736 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:39.523799 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:39.524149 (+ 350us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.524864 (+ 715us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:39.525995 (+ 1131us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.526550 (+ 555us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:39.526605 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:39.526640 (+ 35us) server_negotiation.cc:300] Negotiation successful
0131 03:12:39.526689 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":98,"threads_started":1}
I20260131 03:12:39.526767 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:39.522943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48442 (local address 127.19.84.3:37531)
0131 03:12:39.523217 (+ 274us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:39.523225 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:39.523235 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:12:39.523275 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:39.523278 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:39.523296 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:39.523369 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:39.523781 (+ 412us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.524952 (+ 1171us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:39.526098 (+ 1146us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.526602 (+ 504us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:39.526637 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:39.526663 (+ 26us) server_negotiation.cc:300] Negotiation successful
0131 03:12:39.526701 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":85,"threads_started":1}
I20260131 03:12:39.527415 3033 tablet_service.cc:1505] Processing CreateTablet for tablet bb50b595ea0e4f3da062bd990a6e7cca (DEFAULT_TABLE table=test-table [id=e901d118286c45fbadc37e44ceab74d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:12:39.527743 3033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb50b595ea0e4f3da062bd990a6e7cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:12:39.527781 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:39.523192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45938 (local address 127.19.84.2:46289)
0131 03:12:39.523352 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:39.523360 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:39.523370 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:12:39.523441 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:39.523443 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:39.523460 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:39.523535 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:39.523817 (+ 282us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.524956 (+ 1139us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:39.526904 (+ 1948us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.527476 (+ 572us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:39.527611 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:39.527647 (+ 36us) server_negotiation.cc:300] Negotiation successful
0131 03:12:39.527707 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260131 03:12:39.527700 3305 tablet_service.cc:1505] Processing CreateTablet for tablet bb50b595ea0e4f3da062bd990a6e7cca (DEFAULT_TABLE table=test-table [id=e901d118286c45fbadc37e44ceab74d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:12:39.527992 3305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb50b595ea0e4f3da062bd990a6e7cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:12:39.528278 3169 tablet_service.cc:1505] Processing CreateTablet for tablet bb50b595ea0e4f3da062bd990a6e7cca (DEFAULT_TABLE table=test-table [id=e901d118286c45fbadc37e44ceab74d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:12:39.528529 3169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb50b595ea0e4f3da062bd990a6e7cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:12:39.530267 3388 tablet_bootstrap.cc:492] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35: Bootstrap starting.
I20260131 03:12:39.530623 3389 tablet_bootstrap.cc:492] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0: Bootstrap starting.
I20260131 03:12:39.530902 3388 tablet_bootstrap.cc:654] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35: Neither blocks nor log segments found. Creating new log.
I20260131 03:12:39.531199 3389 tablet_bootstrap.cc:654] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0: Neither blocks nor log segments found. Creating new log.
I20260131 03:12:39.531441 3388 log.cc:826] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:39.531538 3389 log.cc:826] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:39.531847 3392 tablet_bootstrap.cc:492] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9: Bootstrap starting.
I20260131 03:12:39.532196 3388 tablet_bootstrap.cc:492] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35: No bootstrap required, opened a new log
I20260131 03:12:39.532263 3388 ts_tablet_manager.cc:1403] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:12:39.532435 3392 tablet_bootstrap.cc:654] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9: Neither blocks nor log segments found. Creating new log.
I20260131 03:12:39.532717 3392 log.cc:826] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:39.533413 3392 tablet_bootstrap.cc:492] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9: No bootstrap required, opened a new log
I20260131 03:12:39.533481 3392 ts_tablet_manager.cc:1403] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:12:39.533913 3388 raft_consensus.cc:359] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:12:39.534054 3388 raft_consensus.cc:385] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:12:39.534082 3388 raft_consensus.cc:740] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b22a6e7c7344c2db0d2824179e31a35, State: Initialized, Role: FOLLOWER
I20260131 03:12:39.534183 3388 consensus_queue.cc:260] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:12:39.534590 3388 ts_tablet_manager.cc:1434] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:12:39.534754 3371 heartbeater.cc:499] Master 127.19.84.62:44007 was elected leader, sending a full tablet report...
I20260131 03:12:39.535151 3389 tablet_bootstrap.cc:492] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0: No bootstrap required, opened a new log
I20260131 03:12:39.535230 3389 ts_tablet_manager.cc:1403] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260131 03:12:39.535881 3392 raft_consensus.cc:359] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:12:39.536026 3392 raft_consensus.cc:385] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:12:39.536062 3392 raft_consensus.cc:740] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a22130f7cb84aa1a2cf411ddf784ec9, State: Initialized, Role: FOLLOWER
I20260131 03:12:39.536154 3392 consensus_queue.cc:260] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:12:39.536391 3392 ts_tablet_manager.cc:1434] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260131 03:12:39.536451 3235 heartbeater.cc:499] Master 127.19.84.62:44007 was elected leader, sending a full tablet report...
I20260131 03:12:39.537323 3389 raft_consensus.cc:359] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:12:39.537470 3389 raft_consensus.cc:385] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:12:39.537494 3389 raft_consensus.cc:740] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e89581941ff47089b5f0a801f9316a0, State: Initialized, Role: FOLLOWER
I20260131 03:12:39.537582 3389 consensus_queue.cc:260] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:12:39.537819 3389 ts_tablet_manager.cc:1434] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260131 03:12:39.537844 3099 heartbeater.cc:499] Master 127.19.84.62:44007 was elected leader, sending a full tablet report...
W20260131 03:12:39.577024 3236 tablet.cc:2378] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:12:39.642458 3394 raft_consensus.cc:493] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:12:39.642581 3394 raft_consensus.cc:515] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:12:39.642966 3394 leader_election.cc:290] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a22130f7cb84aa1a2cf411ddf784ec9 (127.19.84.2:46289), 0e89581941ff47089b5f0a801f9316a0 (127.19.84.1:43179)
I20260131 03:12:39.643654 3395 raft_consensus.cc:493] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:12:39.643754 3395 raft_consensus.cc:515] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:12:39.644050 3395 leader_election.cc:290] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b22a6e7c7344c2db0d2824179e31a35 (127.19.84.3:37531), 0e89581941ff47089b5f0a801f9316a0 (127.19.84.1:43179)
I20260131 03:12:39.645767 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:39.643135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:46289 (local address 127.19.84.3:59389)
0131 03:12:39.643193 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:39.643206 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:39.643248 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:39.643490 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:39.643491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:39.643494 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:39.643725 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:39.643727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.644673 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:39.644674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:39.645573 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:39.645575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.645633 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:39.645637 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:12:39.645668 (+ 31us) client_negotiation.cc:240] Negotiation successful
0131 03:12:39.645700 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260131 03:12:39.645972 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:39.643135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:43179 (local address 127.19.84.3:40823)
0131 03:12:39.643357 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:39.643365 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:39.643453 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:39.643598 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:39.643599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:39.643601 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:39.643904 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:39.643907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.644826 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:39.644828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:39.645755 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:39.645757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.645819 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:39.645823 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:12:39.645858 (+ 35us) client_negotiation.cc:240] Negotiation successful
0131 03:12:39.645897 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":11,"thread_start_us":137,"threads_started":1}
I20260131 03:12:39.646435 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:39.643247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:59389 (local address 127.19.84.2:46289)
0131 03:12:39.643290 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:39.643292 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:39.643302 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:12:39.643332 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:39.643333 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:39.643349 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:39.643480 (+ 131us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:39.643824 (+ 344us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.644590 (+ 766us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:39.645917 (+ 1327us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.646325 (+ 408us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:39.646333 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:39.646352 (+ 19us) server_negotiation.cc:300] Negotiation successful
0131 03:12:39.646377 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260131 03:12:39.646656 3189 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb50b595ea0e4f3da062bd990a6e7cca" candidate_uuid: "0b22a6e7c7344c2db0d2824179e31a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" is_pre_election: true
I20260131 03:12:39.646721 3387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:39.643247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:40823 (local address 127.19.84.1:43179)
0131 03:12:39.643290 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:39.643292 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:39.643454 (+ 162us) server_negotiation.cc:408] Connection header received
0131 03:12:39.643509 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:39.643510 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:39.643521 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:39.643595 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:39.643994 (+ 399us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.644739 (+ 745us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:39.646132 (+ 1393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.646594 (+ 462us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:39.646601 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:39.646620 (+ 19us) server_negotiation.cc:300] Negotiation successful
0131 03:12:39.646654 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260131 03:12:39.646750 3189 raft_consensus.cc:2468] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b22a6e7c7344c2db0d2824179e31a35 in term 0.
I20260131 03:12:39.646656 3237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:39.644193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:37531 (local address 127.19.84.2:60219)
0131 03:12:39.644315 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:39.644326 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:39.644380 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:39.644567 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:39.644568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:39.644570 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:39.644761 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:39.644762 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.645605 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:39.645606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:39.646449 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:39.646451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.646535 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:39.646539 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:12:39.646574 (+ 35us) client_negotiation.cc:240] Negotiation successful
0131 03:12:39.646609 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260131 03:12:39.646925 3261 leader_election.cc:304] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b22a6e7c7344c2db0d2824179e31a35, 7a22130f7cb84aa1a2cf411ddf784ec9; no voters:
I20260131 03:12:39.647061 3053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb50b595ea0e4f3da062bd990a6e7cca" candidate_uuid: "0b22a6e7c7344c2db0d2824179e31a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e89581941ff47089b5f0a801f9316a0" is_pre_election: true
I20260131 03:12:39.647101 3394 raft_consensus.cc:2804] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:12:39.647168 3394 raft_consensus.cc:493] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:12:39.647176 3053 raft_consensus.cc:2468] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b22a6e7c7344c2db0d2824179e31a35 in term 0.
I20260131 03:12:39.647197 3394 raft_consensus.cc:3060] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:12:39.647250 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:39.644242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:60219 (local address 127.19.84.3:37531)
0131 03:12:39.644276 (+ 34us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:39.644278 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:39.644438 (+ 160us) server_negotiation.cc:408] Connection header received
0131 03:12:39.644457 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:39.644457 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:39.644482 (+ 25us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:39.644522 (+ 40us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:39.644841 (+ 319us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.645513 (+ 672us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:39.646690 (+ 1177us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.647144 (+ 454us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:39.647151 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:39.647173 (+ 22us) server_negotiation.cc:300] Negotiation successful
0131 03:12:39.647203 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260131 03:12:39.647459 3325 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb50b595ea0e4f3da062bd990a6e7cca" candidate_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b22a6e7c7344c2db0d2824179e31a35" is_pre_election: true
I20260131 03:12:39.647804 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:39.644275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:43179 (local address 127.19.84.2:59955)
0131 03:12:39.644655 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:39.644665 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:39.644716 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:39.644928 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:39.644929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:39.644930 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:39.645202 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:39.645204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.646701 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:39.646702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:39.647592 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:39.647595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.647666 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:39.647670 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:12:39.647703 (+ 33us) client_negotiation.cc:240] Negotiation successful
0131 03:12:39.647735 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":79,"threads_started":1}
I20260131 03:12:39.648200 3399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:39.644446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:59955 (local address 127.19.84.1:43179)
0131 03:12:39.644807 (+ 361us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:39.644809 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:39.644823 (+ 14us) server_negotiation.cc:408] Connection header received
0131 03:12:39.644843 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:39.644843 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:39.644851 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:39.645269 (+ 418us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:39.645343 (+ 74us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.646433 (+ 1090us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:39.647666 (+ 1233us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.648113 (+ 447us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:39.648118 (+ 5us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:39.648137 (+ 19us) server_negotiation.cc:300] Negotiation successful
0131 03:12:39.648156 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":82,"threads_started":1}
I20260131 03:12:39.648401 3053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb50b595ea0e4f3da062bd990a6e7cca" candidate_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e89581941ff47089b5f0a801f9316a0" is_pre_election: true
I20260131 03:12:39.648396 3394 raft_consensus.cc:515] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:12:39.648499 3053 raft_consensus.cc:2468] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a22130f7cb84aa1a2cf411ddf784ec9 in term 0.
I20260131 03:12:39.648564 3325 raft_consensus.cc:2393] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a22130f7cb84aa1a2cf411ddf784ec9 in current term 1: Already voted for candidate 0b22a6e7c7344c2db0d2824179e31a35 in this term.
I20260131 03:12:39.648692 3124 leader_election.cc:304] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e89581941ff47089b5f0a801f9316a0, 7a22130f7cb84aa1a2cf411ddf784ec9; no voters:
I20260131 03:12:39.648730 3053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb50b595ea0e4f3da062bd990a6e7cca" candidate_uuid: "0b22a6e7c7344c2db0d2824179e31a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e89581941ff47089b5f0a801f9316a0"
I20260131 03:12:39.648710 3189 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb50b595ea0e4f3da062bd990a6e7cca" candidate_uuid: "0b22a6e7c7344c2db0d2824179e31a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9"
I20260131 03:12:39.648782 3053 raft_consensus.cc:3060] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:12:39.648806 3189 raft_consensus.cc:3060] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:12:39.649557 3053 raft_consensus.cc:2468] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b22a6e7c7344c2db0d2824179e31a35 in term 1.
I20260131 03:12:39.649632 3189 raft_consensus.cc:2468] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b22a6e7c7344c2db0d2824179e31a35 in term 1.
I20260131 03:12:39.649724 3395 raft_consensus.cc:2764] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260131 03:12:39.648720 3394 leader_election.cc:290] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [CANDIDATE]: Term 1 election: Requested vote from peers 7a22130f7cb84aa1a2cf411ddf784ec9 (127.19.84.2:46289), 0e89581941ff47089b5f0a801f9316a0 (127.19.84.1:43179)
I20260131 03:12:39.649737 3260 leader_election.cc:304] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b22a6e7c7344c2db0d2824179e31a35, 0e89581941ff47089b5f0a801f9316a0; no voters:
I20260131 03:12:39.649988 3394 raft_consensus.cc:2804] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:12:39.650163 3394 raft_consensus.cc:697] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [term 1 LEADER]: Becoming Leader. State: Replica: 0b22a6e7c7344c2db0d2824179e31a35, State: Running, Role: LEADER
I20260131 03:12:39.650262 3394 consensus_queue.cc:237] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:12:39.651027 2906 catalog_manager.cc:5654] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b22a6e7c7344c2db0d2824179e31a35 (127.19.84.3). New cstate: current_term: 1 leader_uuid: "0b22a6e7c7344c2db0d2824179e31a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } health_report { overall_health: UNKNOWN } } }
W20260131 03:12:39.659097 3100 tablet.cc:2378] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:12:39.671026 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:39.669069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48446 (local address 127.19.84.3:37531)
0131 03:12:39.669127 (+ 58us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:39.669129 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:39.669156 (+ 27us) server_negotiation.cc:408] Connection header received
0131 03:12:39.669197 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:39.669198 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:39.669215 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:39.669260 (+ 45us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:12:39.669490 (+ 230us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.669920 (+ 430us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:39.670494 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:39.670654 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:39.670688 (+ 34us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:12:39.670774 (+ 86us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:12:39.670838 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:39.670896 (+ 58us) server_negotiation.cc:300] Negotiation successful
0131 03:12:39.670915 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260131 03:12:39.672683 3189 raft_consensus.cc:1275] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 1 FOLLOWER]: Refusing update from remote peer 0b22a6e7c7344c2db0d2824179e31a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:12:39.672683 3053 raft_consensus.cc:1275] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0 [term 1 FOLLOWER]: Refusing update from remote peer 0b22a6e7c7344c2db0d2824179e31a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:12:39.672981 3400 consensus_queue.cc:1048] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:12:39.673110 3394 consensus_queue.cc:1048] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:12:39.675410 3403 mvcc.cc:204] Tried to move back new op lower bound from 7249220238016733184 to 7249220237928173568. Current Snapshot: MvccSnapshot[applied={T|T < 7249220238016733184}]
I20260131 03:12:39.676064 3404 mvcc.cc:204] Tried to move back new op lower bound from 7249220238016733184 to 7249220237928173568. Current Snapshot: MvccSnapshot[applied={T|T < 7249220238016733184}]
I20260131 03:12:39.677655 2904 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:12:39.677759 2904 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:12:39.678542 2904 catalog_manager.cc:5941] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff: Sending DeleteTablet for 3 replicas of tablet bb50b595ea0e4f3da062bd990a6e7cca
I20260131 03:12:39.678876 3169 tablet_service.cc:1552] Processing DeleteTablet for tablet bb50b595ea0e4f3da062bd990a6e7cca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:12:39 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:45938
I20260131 03:12:39.678927 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 2970
I20260131 03:12:39.678911 3033 tablet_service.cc:1552] Processing DeleteTablet for tablet bb50b595ea0e4f3da062bd990a6e7cca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:12:39 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:48342
I20260131 03:12:39.679080 3305 tablet_service.cc:1552] Processing DeleteTablet for tablet bb50b595ea0e4f3da062bd990a6e7cca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:12:39 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:48442
I20260131 03:12:39.680404 3413 tablet_replica.cc:333] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9: stopping tablet replica
I20260131 03:12:39.680507 3413 raft_consensus.cc:2243] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:12:39.680608 3413 raft_consensus.cc:2272] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:12:39.681022 3413 ts_tablet_manager.cc:1916] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:12:39.682642 3413 ts_tablet_manager.cc:1929] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:12:39.682714 3413 log.cc:1199] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/wal/wals/bb50b595ea0e4f3da062bd990a6e7cca
I20260131 03:12:39.683038 3413 ts_tablet_manager.cc:1950] T bb50b595ea0e4f3da062bd990a6e7cca P 7a22130f7cb84aa1a2cf411ddf784ec9: Deleting consensus metadata
I20260131 03:12:39.683456 2893 catalog_manager.cc:4985] TS 7a22130f7cb84aa1a2cf411ddf784ec9 (127.19.84.2:46289): tablet bb50b595ea0e4f3da062bd990a6e7cca (table test-table [id=e901d118286c45fbadc37e44ceab74d2]) successfully deleted
W20260131 03:12:39.684811 2892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.84.1:43179 (error 108)
I20260131 03:12:39.684991 3414 tablet_replica.cc:333] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35: stopping tablet replica
I20260131 03:12:39.685079 3414 raft_consensus.cc:2243] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:12:39.685207 3414 raft_consensus.cc:2272] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:12:39.685397 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 3106
I20260131 03:12:39.685544 3385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:39.685397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:43179 (local address 127.0.0.1:48354)
0131 03:12:39.685463 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:39.685506 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.1:43179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260131 03:12:39.686616 2892 catalog_manager.cc:4712] TS 0e89581941ff47089b5f0a801f9316a0 (127.19.84.1:43179): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb50b595ea0e4f3da062bd990a6e7cca: Network error: Client connection negotiation failed: client connection to 127.19.84.1:43179: connect: Connection refused (error 111)
I20260131 03:12:39.690397 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 3242
I20260131 03:12:39.690428 3414 ts_tablet_manager.cc:1916] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:12:39.695184 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 2874
I20260131 03:12:39.695250 3385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:39.695103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:37531 (local address 127.0.0.1:48452)
0131 03:12:39.695177 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:39.695209 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.3:37531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260131 03:12:59.700803 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:44007
--webserver_interface=127.19.84.62
--webserver_port=45217
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:44007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:12:59.780031 3416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:59.780207 3416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:59.780229 3416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:59.781598 3416 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260131 03:12:59.781634 3416 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260131 03:12:59.781647 3416 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260131 03:12:59.781659 3416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:12:59.781672 3416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:59.781685 3416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:59.781697 3416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:12:59.781708 3416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:12:59.783181 3416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:44007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:44007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=45217
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:59.783354 3416 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:59.783564 3416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:59.789333 3423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:59.789466 3422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:59.789501 3416 server_base.cc:1060] running on GCE node
W20260131 03:12:59.789311 3425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:59.790043 3416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:59.790310 3416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:59.791432 3416 hybrid_clock.cc:648] HybridClock initialized: now 1769829179791404 us; error 47 us; skew 500 ppm
Jan 31 03:12:59 dist-test-slave-5nnp krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829179, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:59.793969 3416 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:12:59.794540 3416 webserver.cc:492] Webserver started at http://127.19.84.62:45217/ using document root <none> and password file <none>
I20260131 03:12:59.794713 3416 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:59.794754 3416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:59.796139 3416 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260131 03:12:59.796813 3432 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:59.797039 3416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:12:59.797134 3416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "ac601929b712426b9fdd57c25bb41aff"
format_stamp: "Formatted at 2026-01-31 03:12:38 on dist-test-slave-5nnp"
server_key: "acb4290367c4772ed1cf87036370060f"
server_key_iv: "a4724bbcd22ec22e928c6ba195f0c29b"
server_key_version: "encryptionkey@0"
I20260131 03:12:59.797473 3416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:59.807137 3416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:59.813058 3416 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:59.813234 3416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:59.817735 3416 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:44007
I20260131 03:12:59.817778 3484 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:44007 every 8 connection(s)
I20260131 03:12:59.818130 3416 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:12:59.819334 3485 sys_catalog.cc:263] Verifying existing consensus state
I20260131 03:12:59.819960 3485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff: Bootstrap starting.
I20260131 03:12:59.825501 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 3416
I20260131 03:12:59.825831 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:43179
--local_ip_for_outbound_sockets=127.19.84.1
--tserver_master_addrs=127.19.84.62:44007
--webserver_port=36309
--webserver_interface=127.19.84.1
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260131 03:12:59.828416 3485 log.cc:826] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:59.830220 3485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260131 03:12:59.830505 3485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff: Bootstrap complete.
I20260131 03:12:59.832532 3485 raft_consensus.cc:359] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac601929b712426b9fdd57c25bb41aff" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44007 } }
I20260131 03:12:59.832743 3485 raft_consensus.cc:740] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac601929b712426b9fdd57c25bb41aff, State: Initialized, Role: FOLLOWER
I20260131 03:12:59.832894 3485 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac601929b712426b9fdd57c25bb41aff" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44007 } }
I20260131 03:12:59.832960 3485 raft_consensus.cc:399] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:12:59.833011 3485 raft_consensus.cc:493] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:12:59.833207 3485 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 1 FOLLOWER]: Advancing to term 2
I20260131 03:12:59.834164 3485 raft_consensus.cc:515] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac601929b712426b9fdd57c25bb41aff" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44007 } }
I20260131 03:12:59.834292 3485 leader_election.cc:304] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac601929b712426b9fdd57c25bb41aff; no voters:
I20260131 03:12:59.834465 3485 leader_election.cc:290] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [CANDIDATE]: Term 2 election: Requested vote from peers
I20260131 03:12:59.834502 3489 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 2 FOLLOWER]: Leader election won for term 2
I20260131 03:12:59.834606 3489 raft_consensus.cc:697] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 2 LEADER]: Becoming Leader. State: Replica: ac601929b712426b9fdd57c25bb41aff, State: Running, Role: LEADER
I20260131 03:12:59.834725 3485 sys_catalog.cc:565] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:12:59.834771 3489 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac601929b712426b9fdd57c25bb41aff" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44007 } }
I20260131 03:12:59.835237 3491 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ac601929b712426b9fdd57c25bb41aff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac601929b712426b9fdd57c25bb41aff" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44007 } } }
I20260131 03:12:59.835327 3491 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [sys.catalog]: This master's current role is: LEADER
I20260131 03:12:59.835580 3490 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac601929b712426b9fdd57c25bb41aff. Latest consensus state: current_term: 2 leader_uuid: "ac601929b712426b9fdd57c25bb41aff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac601929b712426b9fdd57c25bb41aff" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44007 } } }
I20260131 03:12:59.835646 3490 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [sys.catalog]: This master's current role is: LEADER
I20260131 03:12:59.836764 3505 catalog_manager.cc:1269] Loaded cluster ID: 3d3ff55d824846cab7c1c66c4ae6e4b8
I20260131 03:12:59.836813 3505 catalog_manager.cc:1562] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff: loading cluster ID for follower catalog manager: success
I20260131 03:12:59.838070 3505 catalog_manager.cc:1584] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff: acquiring CA information for follower catalog manager: success
I20260131 03:12:59.838438 3505 catalog_manager.cc:1612] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260131 03:12:59.838517 3499 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:12:59.838804 3499 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:12:59.838850 3499 catalog_manager.cc:1269] Loaded cluster ID: 3d3ff55d824846cab7c1c66c4ae6e4b8
I20260131 03:12:59.838865 3499 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:12:59.838956 3499 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:12:59.838995 3499 catalog_manager.cc:6038] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff: Loaded TSK: 0
I20260131 03:12:59.843891 3499 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff: Generated new TSK 1
I20260131 03:12:59.843994 3499 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260131 03:12:59.918283 3487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:12:59.918509 3487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:12:59.918553 3487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:12:59.920136 3487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:12:59.920199 3487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:12:59.920244 3487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:12:59.921860 3487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:43179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=36309
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:44007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:12:59.922108 3487 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:12:59.922308 3487 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:12:59.924786 3515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:59.924786 3512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:12:59.924804 3513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:12:59.925423 3487 server_base.cc:1060] running on GCE node
I20260131 03:12:59.925611 3487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:12:59.925954 3487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:12:59.927115 3487 hybrid_clock.cc:648] HybridClock initialized: now 1769829179927094 us; error 40 us; skew 500 ppm
Jan 31 03:12:59 dist-test-slave-5nnp krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829179, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:12:59.929832 3487 init.cc:377] Logged in from keytab as kudu/127.19.84.1@KRBTEST.COM (short username kudu)
I20260131 03:12:59.930393 3487 webserver.cc:492] Webserver started at http://127.19.84.1:36309/ using document root <none> and password file <none>
I20260131 03:12:59.930619 3487 fs_manager.cc:362] Metadata directory not provided
I20260131 03:12:59.930691 3487 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:12:59.932029 3487 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:12:59.932686 3522 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:12:59.932888 3487 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:12:59.932976 3487 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "0e89581941ff47089b5f0a801f9316a0"
format_stamp: "Formatted at 2026-01-31 03:12:39 on dist-test-slave-5nnp"
server_key: "c2752560b142f91111774841f87be240"
server_key_iv: "f3582a8173abc62bd818f023a3e49f13"
server_key_version: "encryptionkey@0"
I20260131 03:12:59.933241 3487 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:12:59.942816 3487 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:12:59.945128 3487 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:12:59.945304 3487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:12:59.945571 3487 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:12:59.946024 3529 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260131 03:12:59.947211 3487 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260131 03:12:59.947310 3487 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260131 03:12:59.947363 3487 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260131 03:12:59.948014 3487 ts_tablet_manager.cc:616] Registered 1 tablets
I20260131 03:12:59.948068 3487 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260131 03:12:59.948130 3529 tablet_bootstrap.cc:492] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0: Bootstrap starting.
I20260131 03:12:59.956001 3487 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:43179
I20260131 03:12:59.956370 3487 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
I20260131 03:12:59.958521 3529 log.cc:826] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0: Log is configured to *not* fsync() on all Append() calls
I20260131 03:12:59.958830 3636 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:43179 every 8 connection(s)
I20260131 03:12:59.960430 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 3487
I20260131 03:12:59.960760 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:46289
--local_ip_for_outbound_sockets=127.19.84.2
--tserver_master_addrs=127.19.84.62:44007
--webserver_port=35963
--webserver_interface=127.19.84.2
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260131 03:12:59.960953 3529 tablet_bootstrap.cc:492] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260131 03:12:59.961225 3529 tablet_bootstrap.cc:492] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0: Bootstrap complete.
I20260131 03:12:59.962383 3529 ts_tablet_manager.cc:1403] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260131 03:12:59.963312 3529 raft_consensus.cc:359] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:12:59.963770 3529 raft_consensus.cc:740] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e89581941ff47089b5f0a801f9316a0, State: Initialized, Role: FOLLOWER
I20260131 03:12:59.963970 3529 consensus_queue.cc:260] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:12:59.964277 3529 ts_tablet_manager.cc:1434] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
Jan 31 03:12:59 dist-test-slave-5nnp krb5kdc[2858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829179, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:12:59.968122 3639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:59.957055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.1:43401)
0131 03:12:59.957205 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:59.957239 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:59.957682 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:59.959136 (+ 1454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:59.959140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:59.959512 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:12:59.959867 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:59.959872 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:59.961434 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:59.961435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:59.962082 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:59.962085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:59.962213 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:59.962897 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:12:59.962920 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:12:59.964727 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:12:59.966674 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:59.966681 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:59.966683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:59.966894 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:59.967111 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:12:59.967112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:12:59.967113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:12:59.967174 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:12:59.967540 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:12:59.967545 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:12:59.967665 (+ 120us) client_negotiation.cc:769] Sending connection context
0131 03:12:59.967770 (+ 105us) client_negotiation.cc:240] Negotiation successful
0131 03:12:59.967927 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260131 03:12:59.968123 3640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:59.957127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:43401 (local address 127.19.84.62:44007)
0131 03:12:59.958761 (+ 1634us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:59.958772 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:59.958808 (+ 36us) server_negotiation.cc:408] Connection header received
0131 03:12:59.958921 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:59.958926 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:59.958975 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:59.959094 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:12:59.959999 (+ 905us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:59.961289 (+ 1290us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:59.962484 (+ 1195us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:59.962728 (+ 244us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:59.964848 (+ 2120us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:12:59.964873 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:12:59.964874 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:12:59.964897 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:12:59.966568 (+ 1671us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:59.966982 (+ 414us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:59.966984 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:59.966985 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:59.967039 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:12:59.967232 (+ 193us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:12:59.967233 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:12:59.967233 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:12:59.967466 (+ 233us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:12:59.967568 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:59.967825 (+ 257us) server_negotiation.cc:300] Negotiation successful
0131 03:12:59.967938 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1585,"thread_start_us":95,"threads_started":1}
I20260131 03:12:59.968912 3637 heartbeater.cc:344] Connected to a master server at 127.19.84.62:44007
I20260131 03:12:59.969002 3637 heartbeater.cc:461] Registering TS with master...
I20260131 03:12:59.969192 3637 heartbeater.cc:507] Master 127.19.84.62:44007 requested a full tablet report, sending...
I20260131 03:12:59.969794 3449 ts_manager.cc:194] Registered new tserver with Master: 0e89581941ff47089b5f0a801f9316a0 (127.19.84.1:43179)
I20260131 03:12:59.970921 3449 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.1@KRBTEST.COM'} at 127.19.84.1:43401
I20260131 03:12:59.971784 3637 heartbeater.cc:499] Master 127.19.84.62:44007 was elected leader, sending a full tablet report...
I20260131 03:12:59.974180 3644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:59.970313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:43179 (local address 127.0.0.1:54830)
0131 03:12:59.971485 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0131 03:12:59.971513 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0131 03:12:59.971596 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:12:59.971829 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:12:59.971832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:12:59.971834 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:12:59.972294 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:59.972297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:59.973145 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:12:59.973146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:12:59.973751 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:12:59.973753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:12:59.973838 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:59.973864 (+ 26us) client_negotiation.cc:769] Sending connection context
0131 03:12:59.973911 (+ 47us) client_negotiation.cc:240] Negotiation successful
0131 03:12:59.973947 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1145,"thread_start_us":91,"threads_started":1}
I20260131 03:12:59.974562 3645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:12:59.970540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54830 (local address 127.19.84.1:43179)
0131 03:12:59.970689 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0131 03:12:59.970696 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:12:59.971658 (+ 962us) server_negotiation.cc:408] Connection header received
0131 03:12:59.971688 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:12:59.971690 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:12:59.971704 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:12:59.971759 (+ 55us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:12:59.972377 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:59.973067 (+ 690us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:12:59.973847 (+ 780us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:12:59.974359 (+ 512us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:12:59.974403 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0131 03:12:59.974435 (+ 32us) server_negotiation.cc:300] Negotiation successful
0131 03:12:59.974483 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260131 03:12:59.974875 3571 tablet_service.cc:1552] Processing DeleteTablet for tablet bb50b595ea0e4f3da062bd990a6e7cca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:12:39 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:54830
I20260131 03:12:59.975356 3646 tablet_replica.cc:333] stopping tablet replica
I20260131 03:12:59.975526 3646 raft_consensus.cc:2243] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:12:59.975605 3646 raft_consensus.cc:2272] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:12:59.975984 3646 ts_tablet_manager.cc:1916] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:12:59.977959 3646 ts_tablet_manager.cc:1929] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:12:59.978055 3646 log.cc:1199] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/wal/wals/bb50b595ea0e4f3da062bd990a6e7cca
I20260131 03:12:59.978335 3646 ts_tablet_manager.cc:1950] T bb50b595ea0e4f3da062bd990a6e7cca P 0e89581941ff47089b5f0a801f9316a0: Deleting consensus metadata
I20260131 03:12:59.978868 3435 catalog_manager.cc:4985] TS 0e89581941ff47089b5f0a801f9316a0 (127.19.84.1:43179): tablet bb50b595ea0e4f3da062bd990a6e7cca (table test-table [id=e901d118286c45fbadc37e44ceab74d2]) successfully deleted
W20260131 03:13:00.064731 3642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:00.064925 3642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:00.064951 3642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:00.066491 3642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:00.066555 3642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:00.066592 3642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:13:00.068295 3642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:46289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=35963
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:44007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:00.068552 3642 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:00.068765 3642 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:00.071612 3654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:00.071764 3642 server_base.cc:1060] running on GCE node
W20260131 03:13:00.071717 3652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:00.071619 3651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:00.072154 3642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:00.072417 3642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:00.073580 3642 hybrid_clock.cc:648] HybridClock initialized: now 1769829180073565 us; error 26 us; skew 500 ppm
Jan 31 03:13:00 dist-test-slave-5nnp krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829180, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:13:00.076179 3642 init.cc:377] Logged in from keytab as kudu/127.19.84.2@KRBTEST.COM (short username kudu)
I20260131 03:13:00.076730 3642 webserver.cc:492] Webserver started at http://127.19.84.2:35963/ using document root <none> and password file <none>
I20260131 03:13:00.076946 3642 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:00.077015 3642 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:00.078315 3642 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:13:00.078860 3661 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:00.079058 3642 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:00.079134 3642 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "7a22130f7cb84aa1a2cf411ddf784ec9"
format_stamp: "Formatted at 2026-01-31 03:12:39 on dist-test-slave-5nnp"
server_key: "f13125ac5206639bba894a27e60e0b1a"
server_key_iv: "58048bfdca9c92acdd187bda64e024bd"
server_key_version: "encryptionkey@0"
I20260131 03:13:00.079465 3642 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:00.106698 3642 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:00.108899 3642 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:00.109014 3642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:00.109205 3642 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:13:00.109539 3642 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:13:00.109575 3642 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:00.109618 3642 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:13:00.109647 3642 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:00.115579 3642 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:46289
I20260131 03:13:00.115638 3774 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:46289 every 8 connection(s)
I20260131 03:13:00.115957 3642 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
I20260131 03:13:00.117782 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 3642
I20260131 03:13:00.118119 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:37531
--local_ip_for_outbound_sockets=127.19.84.3
--tserver_master_addrs=127.19.84.62:44007
--webserver_port=40797
--webserver_interface=127.19.84.3
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jan 31 03:13:00 dist-test-slave-5nnp krb5kdc[2858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829180, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:13:00.125775 3640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:00.116721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:33599 (local address 127.19.84.62:44007)
0131 03:13:00.116818 (+ 97us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:00.116819 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:00.116977 (+ 158us) server_negotiation.cc:408] Connection header received
0131 03:13:00.117459 (+ 482us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:00.117461 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:00.117472 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:00.117551 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:00.118725 (+ 1174us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.119219 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:00.119899 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.120081 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:00.122472 (+ 2391us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:00.122480 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:00.122480 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:00.122490 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:00.124410 (+ 1920us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:00.124823 (+ 413us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:00.124824 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:00.124825 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:00.124856 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:00.125046 (+ 190us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:00.125047 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:00.125047 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:00.125190 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:00.125267 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:00.125515 (+ 248us) server_negotiation.cc:300] Negotiation successful
0131 03:13:00.125650 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260131 03:13:00.125785 3777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:00.116677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.2:33599)
0131 03:13:00.116837 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:00.116873 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:00.117345 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:00.117658 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:00.117664 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:00.118182 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:13:00.118620 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:00.118626 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.119319 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:00.119320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:00.119798 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:00.119800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.119971 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:00.120985 (+ 1014us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:13:00.121004 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:13:00.122379 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:13:00.124516 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:00.124523 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:00.124526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:00.124745 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:00.124923 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:00.124924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:00.124924 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:00.124985 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:00.125260 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:13:00.125264 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:13:00.125384 (+ 120us) client_negotiation.cc:769] Sending connection context
0131 03:13:00.125475 (+ 91us) client_negotiation.cc:240] Negotiation successful
0131 03:13:00.125620 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260131 03:13:00.126410 3775 heartbeater.cc:344] Connected to a master server at 127.19.84.62:44007
I20260131 03:13:00.126581 3775 heartbeater.cc:461] Registering TS with master...
I20260131 03:13:00.126801 3775 heartbeater.cc:507] Master 127.19.84.62:44007 requested a full tablet report, sending...
I20260131 03:13:00.127197 3449 ts_manager.cc:194] Registered new tserver with Master: 7a22130f7cb84aa1a2cf411ddf784ec9 (127.19.84.2:46289)
I20260131 03:13:00.127643 3449 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.2@KRBTEST.COM'} at 127.19.84.2:33599
W20260131 03:13:00.201308 3778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:00.201555 3778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:00.201583 3778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:00.203215 3778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:00.203266 3778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:00.203296 3778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:13:00.204958 3778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:37531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=40797
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:44007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:00.205204 3778 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:00.205405 3778 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:00.208063 3786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:00.208071 3783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:00.208114 3784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:00.208245 3778 server_base.cc:1060] running on GCE node
I20260131 03:13:00.208397 3778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:00.208652 3778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:00.209820 3778 hybrid_clock.cc:648] HybridClock initialized: now 1769829180209774 us; error 63 us; skew 500 ppm
Jan 31 03:13:00 dist-test-slave-5nnp krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829180, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:13:00.212585 3778 init.cc:377] Logged in from keytab as kudu/127.19.84.3@KRBTEST.COM (short username kudu)
I20260131 03:13:00.213167 3778 webserver.cc:492] Webserver started at http://127.19.84.3:40797/ using document root <none> and password file <none>
I20260131 03:13:00.213368 3778 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:00.213435 3778 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:00.214735 3778 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:13:00.215508 3793 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:00.215678 3778 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:13:00.215761 3778 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "0b22a6e7c7344c2db0d2824179e31a35"
format_stamp: "Formatted at 2026-01-31 03:12:39 on dist-test-slave-5nnp"
server_key: "bb2f7fd82f3e459e95a3466a3241e01b"
server_key_iv: "d7f8bf897298dde5786b318310fe9221"
server_key_version: "encryptionkey@0"
I20260131 03:13:00.216033 3778 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:00.229619 3778 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:00.231640 3778 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:00.231786 3778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:00.231995 3778 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:13:00.232398 3800 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260131 03:13:00.233304 3778 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260131 03:13:00.233359 3778 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260131 03:13:00.233405 3778 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260131 03:13:00.233968 3778 ts_tablet_manager.cc:616] Registered 1 tablets
I20260131 03:13:00.234020 3778 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260131 03:13:00.234090 3800 tablet_bootstrap.cc:492] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35: Bootstrap starting.
I20260131 03:13:00.235260 3800 log.cc:826] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:00.237937 3800 tablet_bootstrap.cc:492] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260131 03:13:00.238214 3800 tablet_bootstrap.cc:492] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35: Bootstrap complete.
I20260131 03:13:00.238353 3800 ts_tablet_manager.cc:1403] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260131 03:13:00.239346 3800 raft_consensus.cc:359] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:13:00.239620 3800 raft_consensus.cc:740] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b22a6e7c7344c2db0d2824179e31a35, State: Initialized, Role: FOLLOWER
I20260131 03:13:00.239768 3800 consensus_queue.cc:260] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:13:00.240031 3800 ts_tablet_manager.cc:1434] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260131 03:13:00.241096 3778 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:37531
I20260131 03:13:00.241205 3909 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:37531 every 8 connection(s)
I20260131 03:13:00.241426 3778 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
I20260131 03:13:00.242933 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 3778
Jan 31 03:13:00 dist-test-slave-5nnp krb5kdc[2858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829180, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:13:00.249140 3640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:00.242076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:56741 (local address 127.19.84.62:44007)
0131 03:13:00.242115 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:00.242117 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:00.242275 (+ 158us) server_negotiation.cc:408] Connection header received
0131 03:13:00.242381 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:00.242382 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:00.242390 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:00.242451 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:00.243228 (+ 777us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.243798 (+ 570us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:00.244317 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.244506 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:00.246434 (+ 1928us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:00.246438 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:00.246438 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:00.246444 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:00.247886 (+ 1442us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:00.248332 (+ 446us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:00.248333 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:00.248333 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:00.248354 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:00.248555 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:00.248556 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:00.248556 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:00.248631 (+ 75us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:00.248712 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:00.248965 (+ 253us) server_negotiation.cc:300] Negotiation successful
0131 03:13:00.249051 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260131 03:13:00.249269 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:00.242057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.3:56741)
0131 03:13:00.242185 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:00.242215 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:00.242331 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:00.242515 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:00.242518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:00.242791 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:13:00.243110 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:00.243113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.243873 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:00.243874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:00.244247 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:00.244249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.244338 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:00.244872 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:13:00.244888 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:13:00.246153 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:13:00.247973 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:00.247980 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:00.247983 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:00.248261 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:00.248415 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:00.248416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:00.248417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:00.248497 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:00.248703 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:13:00.248709 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:13:00.248838 (+ 129us) client_negotiation.cc:769] Sending connection context
0131 03:13:00.248946 (+ 108us) client_negotiation.cc:240] Negotiation successful
0131 03:13:00.249074 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":70,"threads_started":1}
I20260131 03:13:00.249747 3910 heartbeater.cc:344] Connected to a master server at 127.19.84.62:44007
I20260131 03:13:00.249859 3910 heartbeater.cc:461] Registering TS with master...
I20260131 03:13:00.249922 3913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:00.243324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38072 (local address 127.19.84.62:44007)
0131 03:13:00.243460 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:00.243462 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:00.243993 (+ 531us) server_negotiation.cc:408] Connection header received
0131 03:13:00.244078 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:00.244079 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:00.244091 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:00.244157 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:00.245000 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.245458 (+ 458us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:00.246075 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.246260 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:00.247062 (+ 802us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:00.247066 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:00.247067 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:00.247073 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:00.248622 (+ 1549us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:00.248930 (+ 308us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:00.248931 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:00.248931 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:00.248954 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:00.249103 (+ 149us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:00.249103 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:00.249104 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:00.249207 (+ 103us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:00.249352 (+ 145us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:00.249718 (+ 366us) server_negotiation.cc:300] Negotiation successful
0131 03:13:00.249809 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":90,"threads_started":1}
I20260131 03:13:00.250065 3910 heartbeater.cc:507] Master 127.19.84.62:44007 requested a full tablet report, sending...
I20260131 03:13:00.250627 3448 ts_manager.cc:194] Registered new tserver with Master: 0b22a6e7c7344c2db0d2824179e31a35 (127.19.84.3:37531)
I20260131 03:13:00.251468 3448 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:56741
Jan 31 03:13:00 dist-test-slave-5nnp krb5kdc[2858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829179, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for kudu/127.19.84.3@KRBTEST.COM
I20260131 03:13:00.258046 3644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:00.250924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:37531 (local address 127.0.0.1:48258)
0131 03:13:00.250992 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:00.251005 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:00.251055 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:00.251435 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:00.251436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:00.251826 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:13:00.251947 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:00.251949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.252793 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:00.252794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:00.253109 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:00.253110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.253187 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:00.253663 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:13:00.253667 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:13:00.254897 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:13:00.257073 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:00.257079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:00.257081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:00.257253 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:00.257432 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:00.257433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:00.257433 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:00.257465 (+ 32us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:00.257694 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:13:00.257699 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:13:00.257748 (+ 49us) client_negotiation.cc:769] Sending connection context
0131 03:13:00.257843 (+ 95us) client_negotiation.cc:240] Negotiation successful
0131 03:13:00.257951 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260131 03:13:00.258082 3915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:00.251032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48258 (local address 127.19.84.3:37531)
0131 03:13:00.251237 (+ 205us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:00.251244 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:00.251253 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:13:00.251278 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:00.251279 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:00.251290 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:00.251382 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:00.252050 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.252694 (+ 644us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:00.253183 (+ 489us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.253407 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:00.254979 (+ 1572us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:00.254987 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:00.254988 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:00.254998 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:00.256971 (+ 1973us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:00.257329 (+ 358us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:00.257331 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:00.257332 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:00.257368 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:00.257523 (+ 155us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:00.257524 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:00.257524 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:00.257629 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:00.257727 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:00.257883 (+ 156us) server_negotiation.cc:300] Negotiation successful
0131 03:13:00.257972 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":108,"threads_started":1}
I20260131 03:13:00.258446 3843 tablet_service.cc:1552] Processing DeleteTablet for tablet bb50b595ea0e4f3da062bd990a6e7cca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:12:39 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:48258
I20260131 03:13:00.258900 3916 tablet_replica.cc:333] stopping tablet replica
I20260131 03:13:00.258996 3916 raft_consensus.cc:2243] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:13:00.259076 3916 raft_consensus.cc:2272] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:13:00.259430 3916 ts_tablet_manager.cc:1916] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:13:00.261739 3916 ts_tablet_manager.cc:1929] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:13:00.261833 3916 log.cc:1199] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/bb50b595ea0e4f3da062bd990a6e7cca
I20260131 03:13:00.261821 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:13:00.262125 3916 ts_tablet_manager.cc:1950] T bb50b595ea0e4f3da062bd990a6e7cca P 0b22a6e7c7344c2db0d2824179e31a35: Deleting consensus metadata
I20260131 03:13:00.262590 3435 catalog_manager.cc:4985] TS 0b22a6e7c7344c2db0d2824179e31a35 (127.19.84.3:37531): tablet bb50b595ea0e4f3da062bd990a6e7cca (table test-table [id=e901d118286c45fbadc37e44ceab74d2]) successfully deleted
I20260131 03:13:00.264537 3913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:00.262184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38074 (local address 127.19.84.62:44007)
0131 03:13:00.262228 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:00.262230 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:00.262359 (+ 129us) server_negotiation.cc:408] Connection header received
0131 03:13:00.262449 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:00.262450 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:00.262476 (+ 26us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:00.262632 (+ 156us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:13:00.262931 (+ 299us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.263444 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:00.263986 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.264107 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:00.264139 (+ 32us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:13:00.264179 (+ 40us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0131 03:13:00.264470 (+ 291us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38074: token expired
Metrics: {"server-negotiator.queue_time_us":24}
W20260131 03:13:00.264600 3913 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38074: token expired
W20260131 03:13:00.264798 3377 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.19.84.62:44007)
I20260131 03:13:00.272007 3913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:00.265064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38090 (local address 127.19.84.62:44007)
0131 03:13:00.265152 (+ 88us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:00.265155 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:00.265164 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:13:00.265206 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:00.265207 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:00.265221 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:00.265275 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:00.266153 (+ 878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.266960 (+ 807us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:00.267830 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.268044 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:00.269199 (+ 1155us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:00.269202 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:00.269203 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:00.269224 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:00.270915 (+ 1691us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:00.271326 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:00.271327 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:00.271327 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:00.271370 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:00.271568 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:00.271569 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:00.271570 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:00.271652 (+ 82us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:00.271728 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:00.271871 (+ 143us) server_negotiation.cc:300] Negotiation successful
0131 03:13:00.271942 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260131 03:13:00.273447 3448 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:13:00.274812 3448 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:13:00.276790 3843 tablet_service.cc:1505] Processing CreateTablet for tablet 50691ebe9ac44f86ae64ddabe87b486e (DEFAULT_TABLE table=test-table [id=aa1fbdd4b0bf441bba4532b3a159217c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:13:00.276791 3571 tablet_service.cc:1505] Processing CreateTablet for tablet 50691ebe9ac44f86ae64ddabe87b486e (DEFAULT_TABLE table=test-table [id=aa1fbdd4b0bf441bba4532b3a159217c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:13:00.276980 3843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50691ebe9ac44f86ae64ddabe87b486e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:00.277004 3571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50691ebe9ac44f86ae64ddabe87b486e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:00.278487 3529 tablet_bootstrap.cc:492] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0: Bootstrap starting.
I20260131 03:13:00.278519 3800 tablet_bootstrap.cc:492] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35: Bootstrap starting.
I20260131 03:13:00.278857 3529 tablet_bootstrap.cc:654] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:00.278957 3800 tablet_bootstrap.cc:654] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:00.279436 3529 tablet_bootstrap.cc:492] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0: No bootstrap required, opened a new log
I20260131 03:13:00.279496 3529 ts_tablet_manager.cc:1403] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260131 03:13:00.279655 3529 raft_consensus.cc:359] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:13:00.279737 3529 raft_consensus.cc:385] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:00.279774 3529 raft_consensus.cc:740] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e89581941ff47089b5f0a801f9316a0, State: Initialized, Role: FOLLOWER
I20260131 03:13:00.279822 3529 consensus_queue.cc:260] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:13:00.279850 3800 tablet_bootstrap.cc:492] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35: No bootstrap required, opened a new log
I20260131 03:13:00.279894 3800 ts_tablet_manager.cc:1403] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260131 03:13:00.279920 3529 ts_tablet_manager.cc:1434] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:00.279978 3644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:00.276669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:46289 (local address 127.0.0.1:50756)
0131 03:13:00.276725 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:00.276736 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:00.276779 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:00.277227 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:00.277228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:00.277230 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:00.277388 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:00.277389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.278738 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:00.278739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:00.279738 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:00.279741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.279819 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:00.279824 (+ 5us) client_negotiation.cc:769] Sending connection context
0131 03:13:00.279848 (+ 24us) client_negotiation.cc:240] Negotiation successful
0131 03:13:00.279902 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260131 03:13:00.280046 3800 raft_consensus.cc:359] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:13:00.280123 3800 raft_consensus.cc:385] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:00.280169 3800 raft_consensus.cc:740] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b22a6e7c7344c2db0d2824179e31a35, State: Initialized, Role: FOLLOWER
I20260131 03:13:00.280220 3800 consensus_queue.cc:260] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:13:00.280308 3800 ts_tablet_manager.cc:1434] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:00.280386 3910 heartbeater.cc:499] Master 127.19.84.62:44007 was elected leader, sending a full tablet report...
I20260131 03:13:00.280635 3918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:00.276784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50756 (local address 127.19.84.2:46289)
0131 03:13:00.276994 (+ 210us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:00.277002 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:00.277011 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:13:00.277051 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:00.277054 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:00.277071 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:00.277230 (+ 159us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:00.277467 (+ 237us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.278636 (+ 1169us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:00.279869 (+ 1233us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.280393 (+ 524us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:00.280474 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:00.280506 (+ 32us) server_negotiation.cc:300] Negotiation successful
0131 03:13:00.280553 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":120,"threads_started":1}
I20260131 03:13:00.281227 3709 tablet_service.cc:1505] Processing CreateTablet for tablet 50691ebe9ac44f86ae64ddabe87b486e (DEFAULT_TABLE table=test-table [id=aa1fbdd4b0bf441bba4532b3a159217c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:13:00.281540 3709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50691ebe9ac44f86ae64ddabe87b486e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:00.283960 3921 tablet_bootstrap.cc:492] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9: Bootstrap starting.
I20260131 03:13:00.284519 3921 tablet_bootstrap.cc:654] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:00.284785 3921 log.cc:826] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:00.285480 3921 tablet_bootstrap.cc:492] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9: No bootstrap required, opened a new log
I20260131 03:13:00.285578 3921 ts_tablet_manager.cc:1403] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:13:00.286707 3921 raft_consensus.cc:359] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:13:00.286821 3921 raft_consensus.cc:385] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:00.286842 3921 raft_consensus.cc:740] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a22130f7cb84aa1a2cf411ddf784ec9, State: Initialized, Role: FOLLOWER
I20260131 03:13:00.286957 3921 consensus_queue.cc:260] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:13:00.287232 3775 heartbeater.cc:499] Master 127.19.84.62:44007 was elected leader, sending a full tablet report...
I20260131 03:13:00.287437 3921 ts_tablet_manager.cc:1434] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:13:00.321671 3888 raft_consensus.cc:493] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:13:00.321810 3888 raft_consensus.cc:515] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:13:00.322160 3888 leader_election.cc:290] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a22130f7cb84aa1a2cf411ddf784ec9 (127.19.84.2:46289), 0e89581941ff47089b5f0a801f9316a0 (127.19.84.1:43179)
I20260131 03:13:00.324604 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:00.322335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:43179 (local address 127.19.84.3:57347)
0131 03:13:00.322388 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:00.322401 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:00.322464 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:00.322676 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:00.322677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:00.322700 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:00.322961 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:00.322963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.323710 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:00.323711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:00.324328 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:00.324330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.324410 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:00.324451 (+ 41us) client_negotiation.cc:769] Sending connection context
0131 03:13:00.324488 (+ 37us) client_negotiation.cc:240] Negotiation successful
0131 03:13:00.324532 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260131 03:13:00.324719 3924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:00.322353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:46289 (local address 127.19.84.3:50185)
0131 03:13:00.322625 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:00.322636 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:00.322702 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:00.322846 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:00.322847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:00.322848 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:00.323068 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:00.323070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.323885 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:00.323886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:00.324450 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:00.324451 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.324539 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:00.324542 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:13:00.324599 (+ 57us) client_negotiation.cc:240] Negotiation successful
0131 03:13:00.324637 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":53,"thread_start_us":115,"threads_started":1}
I20260131 03:13:00.325053 3645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:00.322452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:57347 (local address 127.19.84.1:43179)
0131 03:13:00.322495 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:00.322496 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:00.322539 (+ 43us) server_negotiation.cc:408] Connection header received
0131 03:13:00.322572 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:00.322573 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:00.322583 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:00.322646 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:00.323051 (+ 405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.323629 (+ 578us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:00.324420 (+ 791us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.324936 (+ 516us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:00.324946 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:00.324970 (+ 24us) server_negotiation.cc:300] Negotiation successful
0131 03:13:00.325008 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260131 03:13:00.325176 3918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:00.322421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:50185 (local address 127.19.84.2:46289)
0131 03:13:00.322472 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:00.322475 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:00.322718 (+ 243us) server_negotiation.cc:408] Connection header received
0131 03:13:00.322751 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:00.322752 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:00.322765 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:00.322838 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:00.323143 (+ 305us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.323747 (+ 604us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:00.324537 (+ 790us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.325029 (+ 492us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:00.325036 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:00.325059 (+ 23us) server_negotiation.cc:300] Negotiation successful
0131 03:13:00.325093 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260131 03:13:00.325361 3591 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50691ebe9ac44f86ae64ddabe87b486e" candidate_uuid: "0b22a6e7c7344c2db0d2824179e31a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e89581941ff47089b5f0a801f9316a0" is_pre_election: true
I20260131 03:13:00.325465 3729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50691ebe9ac44f86ae64ddabe87b486e" candidate_uuid: "0b22a6e7c7344c2db0d2824179e31a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" is_pre_election: true
I20260131 03:13:00.325541 3591 raft_consensus.cc:2468] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b22a6e7c7344c2db0d2824179e31a35 in term 0.
I20260131 03:13:00.325619 3729 raft_consensus.cc:2468] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b22a6e7c7344c2db0d2824179e31a35 in term 0.
I20260131 03:13:00.326026 3796 leader_election.cc:304] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b22a6e7c7344c2db0d2824179e31a35, 0e89581941ff47089b5f0a801f9316a0; no voters:
I20260131 03:13:00.326196 3888 raft_consensus.cc:2804] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:13:00.326269 3888 raft_consensus.cc:493] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:13:00.326316 3888 raft_consensus.cc:3060] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:00.327034 3888 raft_consensus.cc:515] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:13:00.327180 3888 leader_election.cc:290] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [CANDIDATE]: Term 1 election: Requested vote from peers 7a22130f7cb84aa1a2cf411ddf784ec9 (127.19.84.2:46289), 0e89581941ff47089b5f0a801f9316a0 (127.19.84.1:43179)
I20260131 03:13:00.327366 3591 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50691ebe9ac44f86ae64ddabe87b486e" candidate_uuid: "0b22a6e7c7344c2db0d2824179e31a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e89581941ff47089b5f0a801f9316a0"
I20260131 03:13:00.327366 3729 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50691ebe9ac44f86ae64ddabe87b486e" candidate_uuid: "0b22a6e7c7344c2db0d2824179e31a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9"
I20260131 03:13:00.327476 3729 raft_consensus.cc:3060] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:00.327476 3591 raft_consensus.cc:3060] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:00.328253 3591 raft_consensus.cc:2468] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b22a6e7c7344c2db0d2824179e31a35 in term 1.
I20260131 03:13:00.328387 3729 raft_consensus.cc:2468] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b22a6e7c7344c2db0d2824179e31a35 in term 1.
I20260131 03:13:00.328476 3796 leader_election.cc:304] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b22a6e7c7344c2db0d2824179e31a35, 0e89581941ff47089b5f0a801f9316a0; no voters:
I20260131 03:13:00.328613 3888 raft_consensus.cc:2804] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:13:00.328776 3888 raft_consensus.cc:697] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [term 1 LEADER]: Becoming Leader. State: Replica: 0b22a6e7c7344c2db0d2824179e31a35, State: Running, Role: LEADER
I20260131 03:13:00.328855 3888 consensus_queue.cc:237] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:13:00.329622 3449 catalog_manager.cc:5654] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b22a6e7c7344c2db0d2824179e31a35 (127.19.84.3). New cstate: current_term: 1 leader_uuid: "0b22a6e7c7344c2db0d2824179e31a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } health_report { overall_health: UNKNOWN } } }
I20260131 03:13:00.342515 3915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:00.340520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48264 (local address 127.19.84.3:37531)
0131 03:13:00.340580 (+ 60us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:00.340582 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:00.340590 (+ 8us) server_negotiation.cc:408] Connection header received
0131 03:13:00.340651 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:00.340652 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:00.340667 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:00.340718 (+ 51us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:13:00.340934 (+ 216us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.341430 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:00.341954 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:00.342098 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:00.342131 (+ 33us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:13:00.342218 (+ 87us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:13:00.342284 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:00.342329 (+ 45us) server_negotiation.cc:300] Negotiation successful
0131 03:13:00.342381 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260131 03:13:00.344003 3729 raft_consensus.cc:1275] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 1 FOLLOWER]: Refusing update from remote peer 0b22a6e7c7344c2db0d2824179e31a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:13:00.344003 3591 raft_consensus.cc:1275] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [term 1 FOLLOWER]: Refusing update from remote peer 0b22a6e7c7344c2db0d2824179e31a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:13:00.344305 3888 consensus_queue.cc:1048] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:13:00.344435 3925 consensus_queue.cc:1048] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:13:00.346421 3927 mvcc.cc:204] Tried to move back new op lower bound from 7249220322686545920 to 7249220322627514368. Current Snapshot: MvccSnapshot[applied={T|T < 7249220322686545920}]
I20260131 03:13:00.346777 3928 mvcc.cc:204] Tried to move back new op lower bound from 7249220322686545920 to 7249220322627514368. Current Snapshot: MvccSnapshot[applied={T|T < 7249220322686545920}]
I20260131 03:13:00.347764 3929 mvcc.cc:204] Tried to move back new op lower bound from 7249220322686545920 to 7249220322627514368. Current Snapshot: MvccSnapshot[applied={T|T < 7249220322686545920 or (T in {7249220322686545920})}]
I20260131 03:13:00.348675 3448 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:13:00.348767 3448 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:13:00.349447 3448 catalog_manager.cc:5941] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff: Sending DeleteTablet for 3 replicas of tablet 50691ebe9ac44f86ae64ddabe87b486e
I20260131 03:13:00.349758 3571 tablet_service.cc:1552] Processing DeleteTablet for tablet 50691ebe9ac44f86ae64ddabe87b486e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:13:00 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:54830
I20260131 03:13:00.349752 3709 tablet_service.cc:1552] Processing DeleteTablet for tablet 50691ebe9ac44f86ae64ddabe87b486e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:13:00 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:50756
I20260131 03:13:00.349794 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 3487
I20260131 03:13:00.349862 3646 tablet_replica.cc:333] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0: stopping tablet replica
I20260131 03:13:00.349915 3646 raft_consensus.cc:2243] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:13:00.349956 3646 raft_consensus.cc:2272] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:13:00.352041 3843 tablet_service.cc:1552] Processing DeleteTablet for tablet 50691ebe9ac44f86ae64ddabe87b486e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:13:00 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:48258
I20260131 03:13:00.352164 3916 tablet_replica.cc:333] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35: stopping tablet replica
I20260131 03:13:00.352208 3916 raft_consensus.cc:2243] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:13:00.352286 3916 raft_consensus.cc:2272] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:13:00.352644 3916 ts_tablet_manager.cc:1916] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:13:00.354243 3916 ts_tablet_manager.cc:1929] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:13:00.354311 3916 log.cc:1199] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/50691ebe9ac44f86ae64ddabe87b486e
I20260131 03:13:00.354523 3916 ts_tablet_manager.cc:1950] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35: Deleting consensus metadata
I20260131 03:13:00.354928 3435 catalog_manager.cc:4985] TS 0b22a6e7c7344c2db0d2824179e31a35 (127.19.84.3:37531): tablet 50691ebe9ac44f86ae64ddabe87b486e (table test-table [id=aa1fbdd4b0bf441bba4532b3a159217c]) successfully deleted
W20260131 03:13:00.355768 3435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.84.1:43179 (error 108)
I20260131 03:13:00.356069 3936 tablet_replica.cc:333] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9: stopping tablet replica
I20260131 03:13:00.356185 3936 raft_consensus.cc:2243] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:13:00.356256 3644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:00.356142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:43179 (local address 127.0.0.1:54832)
0131 03:13:00.356191 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:00.356223 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.1:43179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260131 03:13:00.356309 3936 raft_consensus.cc:2272] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:13:00.356536 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 3642
W20260131 03:13:00.356348 3435 catalog_manager.cc:4712] TS 0e89581941ff47089b5f0a801f9316a0 (127.19.84.1:43179): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 50691ebe9ac44f86ae64ddabe87b486e: Network error: Client connection negotiation failed: client connection to 127.19.84.1:43179: connect: Connection refused (error 111)
I20260131 03:13:00.361622 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 3778
I20260131 03:13:00.361685 3644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:00.361553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:46289 (local address 127.0.0.1:50762)
0131 03:13:00.361622 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:00.361651 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.2:46289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260131 03:13:00.366880 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 3416
I20260131 03:13:20.373176 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:44007
--webserver_interface=127.19.84.62
--webserver_port=45217
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:44007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:13:20.452962 3938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:20.453176 3938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:20.453210 3938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:20.454658 3938 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260131 03:13:20.454728 3938 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260131 03:13:20.454751 3938 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260131 03:13:20.454770 3938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:13:20.454803 3938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:20.454835 3938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:20.454864 3938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:13:20.454887 3938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:13:20.456646 3938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:44007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:44007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=45217
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:20.456913 3938 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:20.457110 3938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:20.459550 3943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:20.459559 3946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:20.459589 3944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:20.459893 3938 server_base.cc:1060] running on GCE node
I20260131 03:13:20.460106 3938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:20.460361 3938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:20.461534 3938 hybrid_clock.cc:648] HybridClock initialized: now 1769829200461510 us; error 37 us; skew 500 ppm
Jan 31 03:13:20 dist-test-slave-5nnp krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829200, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:13:20.464049 3938 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:13:20.464622 3938 webserver.cc:492] Webserver started at http://127.19.84.62:45217/ using document root <none> and password file <none>
I20260131 03:13:20.464833 3938 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:20.464890 3938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:20.466272 3938 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:13:20.466920 3953 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:20.467123 3938 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:20.467211 3938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "ac601929b712426b9fdd57c25bb41aff"
format_stamp: "Formatted at 2026-01-31 03:12:38 on dist-test-slave-5nnp"
server_key: "acb4290367c4772ed1cf87036370060f"
server_key_iv: "a4724bbcd22ec22e928c6ba195f0c29b"
server_key_version: "encryptionkey@0"
I20260131 03:13:20.467693 3938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:20.476524 3938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:20.484192 3938 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:20.484361 3938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:20.488940 3938 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:44007
I20260131 03:13:20.489022 4005 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:44007 every 8 connection(s)
I20260131 03:13:20.489364 3938 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:13:20.490756 4006 sys_catalog.cc:263] Verifying existing consensus state
I20260131 03:13:20.491492 4006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff: Bootstrap starting.
I20260131 03:13:20.498435 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 3938
I20260131 03:13:20.498850 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:43179
--local_ip_for_outbound_sockets=127.19.84.1
--tserver_master_addrs=127.19.84.62:44007
--webserver_port=36309
--webserver_interface=127.19.84.1
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260131 03:13:20.501313 4006 log.cc:826] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:20.503986 4006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260131 03:13:20.504313 4006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff: Bootstrap complete.
I20260131 03:13:20.506613 4006 raft_consensus.cc:359] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac601929b712426b9fdd57c25bb41aff" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44007 } }
I20260131 03:13:20.506826 4006 raft_consensus.cc:740] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac601929b712426b9fdd57c25bb41aff, State: Initialized, Role: FOLLOWER
I20260131 03:13:20.506978 4006 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac601929b712426b9fdd57c25bb41aff" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44007 } }
I20260131 03:13:20.507052 4006 raft_consensus.cc:399] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:13:20.507121 4006 raft_consensus.cc:493] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:13:20.507172 4006 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 2 FOLLOWER]: Advancing to term 3
I20260131 03:13:20.508209 4006 raft_consensus.cc:515] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac601929b712426b9fdd57c25bb41aff" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44007 } }
I20260131 03:13:20.508306 4006 leader_election.cc:304] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac601929b712426b9fdd57c25bb41aff; no voters:
I20260131 03:13:20.508546 4006 leader_election.cc:290] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [CANDIDATE]: Term 3 election: Requested vote from peers
I20260131 03:13:20.508589 4010 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 3 FOLLOWER]: Leader election won for term 3
I20260131 03:13:20.508704 4010 raft_consensus.cc:697] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [term 3 LEADER]: Becoming Leader. State: Replica: ac601929b712426b9fdd57c25bb41aff, State: Running, Role: LEADER
I20260131 03:13:20.508813 4006 sys_catalog.cc:565] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:13:20.508855 4010 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac601929b712426b9fdd57c25bb41aff" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44007 } }
I20260131 03:13:20.509265 4012 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac601929b712426b9fdd57c25bb41aff. Latest consensus state: current_term: 3 leader_uuid: "ac601929b712426b9fdd57c25bb41aff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac601929b712426b9fdd57c25bb41aff" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44007 } } }
I20260131 03:13:20.509297 4011 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ac601929b712426b9fdd57c25bb41aff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac601929b712426b9fdd57c25bb41aff" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 44007 } } }
I20260131 03:13:20.509358 4012 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [sys.catalog]: This master's current role is: LEADER
I20260131 03:13:20.509380 4011 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff [sys.catalog]: This master's current role is: LEADER
I20260131 03:13:20.509694 4016 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:13:20.510558 4016 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:13:20.510706 4016 catalog_manager.cc:1269] Loaded cluster ID: 3d3ff55d824846cab7c1c66c4ae6e4b8
I20260131 03:13:20.510798 4016 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:13:20.512444 4016 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:13:20.512640 4016 catalog_manager.cc:6038] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff: Loaded TSK: 0
I20260131 03:13:20.512689 4016 catalog_manager.cc:6038] T 00000000000000000000000000000000 P ac601929b712426b9fdd57c25bb41aff: Loaded TSK: 1
F20260131 03:13:20.512902 4016 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6d20ca6dcd google::LogMessage::Fail() at ??:0
@ 0x7f6d20caab93 google::LogMessage::SendToLog() at ??:0
@ 0x7f6d20ca67cc google::LogMessage::Flush() at ??:0
@ 0x7f6d20ca7f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6d1f4e5646 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f6d1f4e2e53 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f6d2353776e kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f6d2353798a kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f6d23537ad3 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f6d234f7597 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f6d23502393 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6d21da8830 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6d21d9e907 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6d220d36db start_thread at ??:0
@ 0x7f6d201fb71f clone at ??:0
W20260131 03:13:20.610095 4008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:20.610363 4008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:20.610405 4008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:20.612941 4008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:20.613029 4008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:20.613073 4008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:13:20.615867 4008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:43179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=36309
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:44007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:20.616127 4008 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:20.616351 4008 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:20.620432 4048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:20.620501 4050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:20.620465 4008 server_base.cc:1060] running on GCE node
W20260131 03:13:20.620414 4047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:20.620759 4008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:20.620988 4008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:20.622155 4008 hybrid_clock.cc:648] HybridClock initialized: now 1769829200622134 us; error 36 us; skew 500 ppm
Jan 31 03:13:20 dist-test-slave-5nnp krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829200, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:13:20.624889 4008 init.cc:377] Logged in from keytab as kudu/127.19.84.1@KRBTEST.COM (short username kudu)
I20260131 03:13:20.625459 4008 webserver.cc:492] Webserver started at http://127.19.84.1:36309/ using document root <none> and password file <none>
I20260131 03:13:20.625671 4008 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:20.625730 4008 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:20.627035 4008 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:13:20.627821 4057 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:20.628008 4008 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260131 03:13:20.628094 4008 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "0e89581941ff47089b5f0a801f9316a0"
format_stamp: "Formatted at 2026-01-31 03:12:39 on dist-test-slave-5nnp"
server_key: "c2752560b142f91111774841f87be240"
server_key_iv: "f3582a8173abc62bd818f023a3e49f13"
server_key_version: "encryptionkey@0"
I20260131 03:13:20.628423 4008 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:20.639605 4008 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:20.641633 4008 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:20.641789 4008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:20.642006 4008 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:13:20.642442 4064 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260131 03:13:20.643616 4008 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260131 03:13:20.643683 4008 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260131 03:13:20.643731 4008 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260131 03:13:20.644393 4008 ts_tablet_manager.cc:616] Registered 1 tablets
I20260131 03:13:20.644449 4008 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260131 03:13:20.644528 4064 tablet_bootstrap.cc:492] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0: Bootstrap starting.
I20260131 03:13:20.652043 4008 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:43179
I20260131 03:13:20.652081 4171 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:43179 every 8 connection(s)
I20260131 03:13:20.652494 4008 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
I20260131 03:13:20.654397 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 4008
I20260131 03:13:20.654726 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:46289
--local_ip_for_outbound_sockets=127.19.84.2
--tserver_master_addrs=127.19.84.62:44007
--webserver_port=35963
--webserver_interface=127.19.84.2
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260131 03:13:20.655550 4064 log.cc:826] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:20.658079 4064 tablet_bootstrap.cc:492] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260131 03:13:20.658421 4064 tablet_bootstrap.cc:492] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0: Bootstrap complete.
I20260131 03:13:20.659689 4064 ts_tablet_manager.cc:1403] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
I20260131 03:13:20.660912 4064 raft_consensus.cc:359] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:13:20.661211 4064 raft_consensus.cc:740] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e89581941ff47089b5f0a801f9316a0, State: Initialized, Role: FOLLOWER
I20260131 03:13:20.661360 4064 consensus_queue.cc:260] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:13:20.661638 4064 ts_tablet_manager.cc:1434] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20260131 03:13:20.741664 4174 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:13:20.653250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.1:43627)
0131 03:13:20.653410 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:20.653443 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:20.653902 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:20.741487 (+ 87585us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
W20260131 03:13:20.741681 4176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:20.741925 4176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:20.742004 4176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:20.742034 4172 heartbeater.cc:646] Failed to heartbeat to 127.19.84.62:44007 (0 consecutive failures): Network error: Failed to ping master at 127.19.84.62:44007: Client connection negotiation failed: client connection to 127.19.84.62:44007: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260131 03:13:20.742550 4174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:20.742284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.1:41481)
0131 03:13:20.742442 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:20.742518 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260131 03:13:20.742857 4174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:20.742782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.1:34087)
0131 03:13:20.742816 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:20.742834 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260131 03:13:20.742951 4172 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260131 03:13:20.744556 4176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:20.744656 4176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:20.744729 4176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:13:20.747321 4176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:46289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=35963
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:44007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:20.747763 4176 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:20.748013 4176 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:20.751227 4182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:20.752254 4183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:20.752779 4185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:20.752890 4176 server_base.cc:1060] running on GCE node
I20260131 03:13:20.753192 4176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:20.753477 4176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:20.754669 4176 hybrid_clock.cc:648] HybridClock initialized: now 1769829200754665 us; error 52 us; skew 500 ppm
Jan 31 03:13:20 dist-test-slave-5nnp krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829200, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:13:20.757424 4176 init.cc:377] Logged in from keytab as kudu/127.19.84.2@KRBTEST.COM (short username kudu)
I20260131 03:13:20.758013 4176 webserver.cc:492] Webserver started at http://127.19.84.2:35963/ using document root <none> and password file <none>
I20260131 03:13:20.758220 4176 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:20.758282 4176 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:20.759641 4176 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:13:20.760273 4192 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:20.760448 4176 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260131 03:13:20.760535 4176 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "7a22130f7cb84aa1a2cf411ddf784ec9"
format_stamp: "Formatted at 2026-01-31 03:12:39 on dist-test-slave-5nnp"
server_key: "f13125ac5206639bba894a27e60e0b1a"
server_key_iv: "58048bfdca9c92acdd187bda64e024bd"
server_key_version: "encryptionkey@0"
I20260131 03:13:20.760807 4176 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:20.782913 4176 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:20.784852 4176 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:20.785009 4176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:20.785246 4176 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:13:20.785704 4199 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260131 03:13:20.786593 4176 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260131 03:13:20.786661 4176 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260131 03:13:20.786732 4176 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260131 03:13:20.787309 4176 ts_tablet_manager.cc:616] Registered 1 tablets
I20260131 03:13:20.787364 4176 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260131 03:13:20.787480 4199 tablet_bootstrap.cc:492] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9: Bootstrap starting.
I20260131 03:13:20.794365 4176 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:46289
I20260131 03:13:20.794462 4306 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:46289 every 8 connection(s)
I20260131 03:13:20.794767 4176 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
I20260131 03:13:20.795835 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:20.795500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.2:38785)
0131 03:13:20.795667 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:20.795729 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
W20260131 03:13:20.796042 4307 heartbeater.cc:646] Failed to heartbeat to 127.19.84.62:44007 (0 consecutive failures): Network error: Failed to ping master at 127.19.84.62:44007: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
I20260131 03:13:20.796392 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:20.796296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.2:51659)
0131 03:13:20.796349 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:20.796374 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260131 03:13:20.796669 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:20.796616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.2:36341)
0131 03:13:20.796640 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:20.796657 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260131 03:13:20.796773 4307 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260131 03:13:20.799300 4199 log.cc:826] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:20.801290 4199 tablet_bootstrap.cc:492] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260131 03:13:20.801578 4199 tablet_bootstrap.cc:492] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9: Bootstrap complete.
I20260131 03:13:20.802546 4199 ts_tablet_manager.cc:1403] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260131 03:13:20.803411 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 4176
I20260131 03:13:20.803728 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:37531
--local_ip_for_outbound_sockets=127.19.84.3
--tserver_master_addrs=127.19.84.62:44007
--webserver_port=40797
--webserver_interface=127.19.84.3
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260131 03:13:20.803912 4199 raft_consensus.cc:359] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:13:20.804239 4199 raft_consensus.cc:740] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a22130f7cb84aa1a2cf411ddf784ec9, State: Initialized, Role: FOLLOWER
I20260131 03:13:20.804348 4199 consensus_queue.cc:260] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:13:20.804615 4199 ts_tablet_manager.cc:1434] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260131 03:13:20.805006 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:20.804863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.2:57801)
0131 03:13:20.804928 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:20.804970 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260131 03:13:20.805413 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:20.805309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.2:41355)
0131 03:13:20.805365 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:20.805395 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260131 03:13:20.882745 4312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:20.882936 4312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:20.882961 4312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:20.884387 4312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:20.884433 4312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:20.884465 4312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:13:20.885977 4312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:37531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=40797
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:44007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:20.886157 4312 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:20.886307 4312 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:20.888873 4317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:20.888895 4320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:20.888938 4318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:20.889393 4312 server_base.cc:1060] running on GCE node
I20260131 03:13:20.889552 4312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:20.889731 4312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:20.890872 4312 hybrid_clock.cc:648] HybridClock initialized: now 1769829200890858 us; error 29 us; skew 500 ppm
Jan 31 03:13:20 dist-test-slave-5nnp krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829200, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:13:20.893483 4312 init.cc:377] Logged in from keytab as kudu/127.19.84.3@KRBTEST.COM (short username kudu)
I20260131 03:13:20.894050 4312 webserver.cc:492] Webserver started at http://127.19.84.3:40797/ using document root <none> and password file <none>
I20260131 03:13:20.894243 4312 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:20.894279 4312 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:20.895570 4312 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:13:20.896330 4327 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:20.896642 4312 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:13:20.896711 4312 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "0b22a6e7c7344c2db0d2824179e31a35"
format_stamp: "Formatted at 2026-01-31 03:12:39 on dist-test-slave-5nnp"
server_key: "bb2f7fd82f3e459e95a3466a3241e01b"
server_key_iv: "d7f8bf897298dde5786b318310fe9221"
server_key_version: "encryptionkey@0"
I20260131 03:13:20.897015 4312 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:20.924899 4312 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:20.927086 4312 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:20.927233 4312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:20.927536 4312 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:13:20.927865 4312 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:13:20.927922 4312 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:20.927963 4312 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:13:20.928004 4312 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:20.934942 4312 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:37531
I20260131 03:13:20.935027 4440 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:37531 every 8 connection(s)
I20260131 03:13:20.935391 4312 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
I20260131 03:13:20.936422 4443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:20.936093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.3:58277)
0131 03:13:20.936225 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:20.936306 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
W20260131 03:13:20.936700 4441 heartbeater.cc:646] Failed to heartbeat to 127.19.84.62:44007 (0 consecutive failures): Network error: Failed to ping master at 127.19.84.62:44007: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
I20260131 03:13:20.937086 4443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:20.936960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.3:36439)
0131 03:13:20.937030 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:20.937063 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260131 03:13:20.937582 4443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:20.937461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.3:36261)
0131 03:13:20.937522 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:20.937552 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260131 03:13:20.937781 4441 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260131 03:13:20.938824 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 4312
I20260131 03:13:21.743605 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:21.743325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.1:50105)
0131 03:13:21.743513 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:21.743558 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":109,"threads_started":1}
I20260131 03:13:21.806161 4446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:21.805914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.2:58713)
0131 03:13:21.806058 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:21.806102 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260131 03:13:21.938447 4447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:21.938198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.3:54635)
0131 03:13:21.938353 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:21.938392 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
I20260131 03:13:22.017519 4448 raft_consensus.cc:493] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:13:22.017695 4448 raft_consensus.cc:515] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:13:22.018045 4448 leader_election.cc:290] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a22130f7cb84aa1a2cf411ddf784ec9 (127.19.84.2:46289), 0b22a6e7c7344c2db0d2824179e31a35 (127.19.84.3:37531)
Jan 31 03:13:22 dist-test-slave-5nnp krb5kdc[2858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829200, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.2@KRBTEST.COM
Jan 31 03:13:22 dist-test-slave-5nnp krb5kdc[2858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829200, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.3@KRBTEST.COM
I20260131 03:13:22.027825 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:22.018232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:46289 (local address 127.19.84.1:35875)
0131 03:13:22.018314 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:22.018327 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:22.018390 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:22.018889 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:22.018892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:22.019411 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:13:22.019753 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:22.019757 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:22.020783 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:22.020784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:22.021328 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:22.021329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:22.021464 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:22.022115 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:13:22.022147 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:13:22.023716 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:13:22.026088 (+ 2372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:22.026097 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:22.026121 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:22.026450 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:22.026734 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:22.026735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:22.026735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:22.026832 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:22.027242 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:13:22.027247 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:13:22.027432 (+ 185us) client_negotiation.cc:769] Sending connection context
0131 03:13:22.027538 (+ 106us) client_negotiation.cc:240] Negotiation successful
0131 03:13:22.027693 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260131 03:13:22.028019 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:22.018240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:37531 (local address 127.19.84.1:43859)
0131 03:13:22.018423 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:22.018436 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:22.018493 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:22.019550 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:22.019552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:22.019717 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:13:22.019935 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:22.019937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:22.021437 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:22.021438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:22.021953 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:22.021956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:22.022022 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:22.022460 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:13:22.022465 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:13:22.024083 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:13:22.026560 (+ 2477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:22.026561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:22.026562 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:22.026808 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:22.027056 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:22.027057 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:22.027058 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:22.027090 (+ 32us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:22.027594 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:13:22.027595 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:13:22.027647 (+ 52us) client_negotiation.cc:769] Sending connection context
0131 03:13:22.027715 (+ 68us) client_negotiation.cc:240] Negotiation successful
0131 03:13:22.027824 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":10,"thread_start_us":70,"threads_started":1}
I20260131 03:13:22.028196 4451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:22.018389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:43859 (local address 127.19.84.3:37531)
0131 03:13:22.018540 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:22.018556 (+ 16us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:22.018605 (+ 49us) server_negotiation.cc:408] Connection header received
0131 03:13:22.019184 (+ 579us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:22.019188 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:22.019248 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:22.019363 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:22.020043 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:22.021317 (+ 1274us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:22.022039 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:22.022260 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:22.024206 (+ 1946us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:22.024212 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:22.024213 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:22.024230 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:22.026327 (+ 2097us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:22.026911 (+ 584us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:22.026914 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:22.026914 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:22.026985 (+ 71us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:22.027153 (+ 168us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:22.027154 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:22.027154 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:22.027454 (+ 300us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:22.027561 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:22.027931 (+ 370us) server_negotiation.cc:300] Negotiation successful
0131 03:13:22.028060 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260131 03:13:22.028631 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:22.018388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:35875 (local address 127.19.84.2:46289)
0131 03:13:22.018518 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:22.018527 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:22.018549 (+ 22us) server_negotiation.cc:408] Connection header received
0131 03:13:22.018668 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:22.018670 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:22.018728 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:22.018847 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:22.019861 (+ 1014us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:22.020682 (+ 821us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:22.021417 (+ 735us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:22.021666 (+ 249us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:22.023842 (+ 2176us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:22.023850 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:22.023850 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:22.023886 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:22.025973 (+ 2087us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:22.026560 (+ 587us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:22.026564 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:22.026565 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:22.026650 (+ 85us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:22.026913 (+ 263us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:22.026913 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:22.026914 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:22.027167 (+ 253us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:22.028147 (+ 980us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:22.028231 (+ 84us) server_negotiation.cc:300] Negotiation successful
0131 03:13:22.028391 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":60,"threads_started":1}
I20260131 03:13:22.035338 4261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50691ebe9ac44f86ae64ddabe87b486e" candidate_uuid: "0e89581941ff47089b5f0a801f9316a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" is_pre_election: true
I20260131 03:13:22.035562 4261 raft_consensus.cc:2468] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e89581941ff47089b5f0a801f9316a0 in term 1.
I20260131 03:13:22.035887 4061 leader_election.cc:304] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e89581941ff47089b5f0a801f9316a0, 7a22130f7cb84aa1a2cf411ddf784ec9; no voters:
W20260131 03:13:22.035995 4060 leader_election.cc:343] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b22a6e7c7344c2db0d2824179e31a35 (127.19.84.3:37531): Not found: Tablet not found: 50691ebe9ac44f86ae64ddabe87b486e
I20260131 03:13:22.036293 4448 raft_consensus.cc:2804] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260131 03:13:22.036374 4448 raft_consensus.cc:493] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:13:22.036401 4448 raft_consensus.cc:3060] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [term 1 FOLLOWER]: Advancing to term 2
I20260131 03:13:22.038050 4448 raft_consensus.cc:515] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:13:22.038213 4448 leader_election.cc:290] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [CANDIDATE]: Term 2 election: Requested vote from peers 7a22130f7cb84aa1a2cf411ddf784ec9 (127.19.84.2:46289), 0b22a6e7c7344c2db0d2824179e31a35 (127.19.84.3:37531)
W20260131 03:13:22.038597 4060 leader_election.cc:343] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 0b22a6e7c7344c2db0d2824179e31a35 (127.19.84.3:37531): Not found: Tablet not found: 50691ebe9ac44f86ae64ddabe87b486e
I20260131 03:13:22.038592 4261 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50691ebe9ac44f86ae64ddabe87b486e" candidate_uuid: "0e89581941ff47089b5f0a801f9316a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9"
I20260131 03:13:22.038695 4261 raft_consensus.cc:3060] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 1 FOLLOWER]: Advancing to term 2
I20260131 03:13:22.040485 4261 raft_consensus.cc:2468] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e89581941ff47089b5f0a801f9316a0 in term 2.
I20260131 03:13:22.040766 4061 leader_election.cc:304] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0e89581941ff47089b5f0a801f9316a0, 7a22130f7cb84aa1a2cf411ddf784ec9; no voters: 0b22a6e7c7344c2db0d2824179e31a35
I20260131 03:13:22.040864 4448 raft_consensus.cc:2804] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [term 2 FOLLOWER]: Leader election won for term 2
I20260131 03:13:22.041008 4448 raft_consensus.cc:697] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [term 2 LEADER]: Becoming Leader. State: Replica: 0e89581941ff47089b5f0a801f9316a0, State: Running, Role: LEADER
I20260131 03:13:22.041111 4448 consensus_queue.cc:237] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:13:22.041980 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:22.041850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.1:56827)
0131 03:13:22.041892 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:22.041945 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260131 03:13:22.470841 4261 raft_consensus.cc:1275] T 50691ebe9ac44f86ae64ddabe87b486e P 7a22130f7cb84aa1a2cf411ddf784ec9 [term 2 FOLLOWER]: Refusing update from remote peer 0e89581941ff47089b5f0a801f9316a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260131 03:13:22.471278 4448 consensus_queue.cc:1048] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260131 03:13:22.472582 4456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:22.472251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.2:59857)
0131 03:13:22.472420 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:22.472511 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":59,"threads_started":1}
I20260131 03:13:22.473366 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:22.473185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.1:44757)
0131 03:13:22.473253 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:22.473287 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260131 03:13:22.473381 4456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:22.473246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.2:57009)
0131 03:13:22.473317 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:22.473353 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260131 03:13:22.473568 4060 consensus_peers.cc:597] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0 -> Peer 0b22a6e7c7344c2db0d2824179e31a35 (127.19.84.3:37531): Couldn't send request to peer 0b22a6e7c7344c2db0d2824179e31a35. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50691ebe9ac44f86ae64ddabe87b486e. This is attempt 1: this message will repeat every 5th retry.
I20260131 03:13:22.939201 4459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:22.938968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.3:53307)
0131 03:13:22.939117 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:22.939148 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260131 03:13:23.083281 4460 ts_tablet_manager.cc:933] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35: Initiating tablet copy from peer 0e89581941ff47089b5f0a801f9316a0 (127.19.84.1:43179)
I20260131 03:13:23.083590 4460 tablet_copy_client.cc:323] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35: tablet copy: Beginning tablet copy session from remote peer at address 127.19.84.1:43179
Jan 31 03:13:23 dist-test-slave-5nnp krb5kdc[2858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829200, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for kudu/127.19.84.1@KRBTEST.COM
I20260131 03:13:23.090791 4459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:23.083874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:43179 (local address 127.19.84.3:39317)
0131 03:13:23.083934 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:23.083950 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:23.084016 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:23.084235 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:23.084238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:23.084581 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:13:23.084755 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:23.084757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:23.085550 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:23.085551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:23.085897 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:23.085898 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:23.085957 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:23.086442 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:13:23.086447 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:13:23.087887 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:13:23.089965 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:23.089970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:23.089972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:23.090133 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:23.090300 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:23.090301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:23.090301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:23.090326 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:23.090527 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:13:23.090531 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:13:23.090572 (+ 41us) client_negotiation.cc:769] Sending connection context
0131 03:13:23.090641 (+ 69us) client_negotiation.cc:240] Negotiation successful
0131 03:13:23.090714 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260131 03:13:23.090806 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:23.083966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:39317 (local address 127.19.84.1:43179)
0131 03:13:23.084101 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:23.084105 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:23.084116 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:13:23.084139 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:23.084141 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:23.084153 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:23.084223 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:23.084836 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:23.085478 (+ 642us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:23.085964 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:23.086134 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:23.088056 (+ 1922us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:23.088065 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:23.088066 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:23.088072 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:23.089847 (+ 1775us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:23.090206 (+ 359us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:23.090208 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:23.090209 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:23.090242 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:23.090374 (+ 132us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:23.090375 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:23.090375 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:23.090476 (+ 101us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:23.090544 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:23.090673 (+ 129us) server_negotiation.cc:300] Negotiation successful
0131 03:13:23.090740 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
I20260131 03:13:23.091193 4146 tablet_copy_service.cc:140] P 0e89581941ff47089b5f0a801f9316a0: Received BeginTabletCopySession request for tablet 50691ebe9ac44f86ae64ddabe87b486e from peer 0b22a6e7c7344c2db0d2824179e31a35 ({username='kudu', principal='kudu/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:39317)
I20260131 03:13:23.091253 4146 tablet_copy_service.cc:161] P 0e89581941ff47089b5f0a801f9316a0: Beginning new tablet copy session on tablet 50691ebe9ac44f86ae64ddabe87b486e from peer 0b22a6e7c7344c2db0d2824179e31a35 at {username='kudu', principal='kudu/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:39317: session id = 0b22a6e7c7344c2db0d2824179e31a35-50691ebe9ac44f86ae64ddabe87b486e
I20260131 03:13:23.091802 4146 tablet_copy_source_session.cc:215] T 50691ebe9ac44f86ae64ddabe87b486e P 0e89581941ff47089b5f0a801f9316a0: Tablet Copy: opened 0 blocks and 1 log segments
I20260131 03:13:23.092660 4460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50691ebe9ac44f86ae64ddabe87b486e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:23.096699 4460 tablet_copy_client.cc:806] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35: tablet copy: Starting download of 0 data blocks...
I20260131 03:13:23.096858 4460 tablet_copy_client.cc:670] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35: tablet copy: Starting download of 1 WAL segments...
I20260131 03:13:23.097910 4460 tablet_copy_client.cc:538] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260131 03:13:23.098588 4460 tablet_bootstrap.cc:492] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35: Bootstrap starting.
I20260131 03:13:23.099880 4460 log.cc:826] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:23.101658 4460 tablet_bootstrap.cc:492] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260131 03:13:23.101876 4460 tablet_bootstrap.cc:492] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35: Bootstrap complete.
I20260131 03:13:23.102005 4460 ts_tablet_manager.cc:1403] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260131 03:13:23.102943 4460 raft_consensus.cc:359] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:13:23.103103 4460 raft_consensus.cc:740] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b22a6e7c7344c2db0d2824179e31a35, State: Initialized, Role: FOLLOWER
I20260131 03:13:23.103219 4460 consensus_queue.cc:260] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a22130f7cb84aa1a2cf411ddf784ec9" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 46289 } } peers { permanent_uuid: "0b22a6e7c7344c2db0d2824179e31a35" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 37531 } } peers { permanent_uuid: "0e89581941ff47089b5f0a801f9316a0" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 43179 } }
I20260131 03:13:23.103526 4460 ts_tablet_manager.cc:1434] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35: Time spent starting tablet: real 0.001s user 0.001s sys 0.002s
I20260131 03:13:23.103893 4459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:23.103758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.3:51961)
0131 03:13:23.103826 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:23.103869 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260131 03:13:23.103880 4146 tablet_copy_service.cc:342] P 0e89581941ff47089b5f0a801f9316a0: Request end of tablet copy session 0b22a6e7c7344c2db0d2824179e31a35-50691ebe9ac44f86ae64ddabe87b486e received from {username='kudu', principal='kudu/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:39317
I20260131 03:13:23.103950 4146 tablet_copy_service.cc:434] P 0e89581941ff47089b5f0a801f9316a0: ending tablet copy session 0b22a6e7c7344c2db0d2824179e31a35-50691ebe9ac44f86ae64ddabe87b486e on tablet 50691ebe9ac44f86ae64ddabe87b486e with peer 0b22a6e7c7344c2db0d2824179e31a35
I20260131 03:13:23.104223 4459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:23.104140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.3:54259)
0131 03:13:23.104173 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:23.104198 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260131 03:13:23.474189 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:23.473961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.1:43753)
0131 03:13:23.474117 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:23.474145 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":74,"threads_started":1}
I20260131 03:13:23.474184 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:23.473961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.2:38895)
0131 03:13:23.474090 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:23.474133 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260131 03:13:23.520689 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:23.520567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.1:58627)
0131 03:13:23.520613 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:23.520649 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260131 03:13:23.520705 4395 raft_consensus.cc:1217] T 50691ebe9ac44f86ae64ddabe87b486e P 0b22a6e7c7344c2db0d2824179e31a35 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260131 03:13:23.521188 4459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:23.521088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.3:33271)
0131 03:13:23.521124 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:23.521156 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260131 03:13:24.474897 4469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:24.474642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.2:36049)
0131 03:13:24.474805 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:24.474848 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":62,"threads_started":1}
I20260131 03:13:24.521443 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:24.521186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.1:54931)
0131 03:13:24.521317 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:24.521369 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260131 03:13:24.521845 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:24.521650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.3:49373)
0131 03:13:24.521774 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:24.521803 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260131 03:13:25.475651 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:25.475432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.2:50633)
0131 03:13:25.475570 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:25.475600 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260131 03:13:25.522192 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:25.521931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.1:49307)
0131 03:13:25.522094 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:25.522142 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
I20260131 03:13:25.522536 4475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:25.522314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.3:56913)
0131 03:13:25.522460 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:25.522493 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
I20260131 03:13:26.476477 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:26.476203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.2:55281)
0131 03:13:26.476354 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:26.476397 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260131 03:13:26.523089 4477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:26.522784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.1:51363)
0131 03:13:26.522983 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:26.523020 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":108,"threads_started":1}
I20260131 03:13:26.523347 4478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:26.523013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.3:57179)
0131 03:13:26.523134 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:26.523173 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260131 03:13:26.523712 4477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:26.523587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.1:48645)
0131 03:13:26.523644 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:26.523678 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260131 03:13:27.477106 4479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:27.476905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.2:38401)
0131 03:13:27.477024 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:27.477060 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260131 03:13:27.524130 4480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:27.523863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.3:37967)
0131 03:13:27.524029 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:27.524079 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":80,"threads_started":1}
I20260131 03:13:27.524339 4481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:27.524146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.1:39517)
0131 03:13:27.524257 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:27.524290 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":65,"threads_started":1}
I20260131 03:13:28.477892 4482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:28.477662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.2:46771)
0131 03:13:28.477799 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:28.477842 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260131 03:13:28.524978 4483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:28.524686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.3:46407)
0131 03:13:28.524868 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:28.524912 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":89,"threads_started":1}
I20260131 03:13:28.525091 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:28.524865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.1:50305)
0131 03:13:28.524986 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:28.525018 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260131 03:13:29.478693 4485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:29.478452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.2:56887)
0131 03:13:29.478599 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:29.478642 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260131 03:13:29.525890 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:29.525608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.1:37495)
0131 03:13:29.525788 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:29.525834 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":90,"threads_started":1}
I20260131 03:13:29.525890 4486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:29.525602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.3:54735)
0131 03:13:29.525795 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:29.525826 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":115,"threads_started":1}
I20260131 03:13:30.479360 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:30.479144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.2:58943)
0131 03:13:30.479278 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:30.479311 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260131 03:13:30.526708 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:30.526441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.1:38387)
0131 03:13:30.526621 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:30.526660 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":116,"threads_started":1}
I20260131 03:13:30.526710 4489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:30.526436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.3:46791)
0131 03:13:30.526630 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:30.526665 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":70,"threads_started":1}
I20260131 03:13:31.480211 4491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:31.479968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.2:35645)
0131 03:13:31.480115 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:31.480159 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
I20260131 03:13:31.527496 4492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:31.527228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.1:53907)
0131 03:13:31.527405 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:31.527449 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":60,"threads_started":1}
I20260131 03:13:31.527532 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:31.527255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.3:53307)
0131 03:13:31.527456 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:31.527489 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":129,"threads_started":1}
I20260131 03:13:32.480973 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:32.480739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.2:53893)
0131 03:13:32.480875 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:32.480913 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
I20260131 03:13:32.528293 4495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:32.528015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.1:51043)
0131 03:13:32.528191 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:32.528236 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":77,"threads_started":1}
I20260131 03:13:32.528293 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:32.528015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.3:39957)
0131 03:13:32.528191 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:32.528236 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":76,"threads_started":1}
I20260131 03:13:33.481951 4497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:33.481559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.2:32883)
0131 03:13:33.481724 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:33.481763 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":68,"threads_started":1}
I20260131 03:13:33.529117 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:33.528878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.1:57279)
0131 03:13:33.529031 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:33.529069 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260131 03:13:33.529119 4498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:33.528859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.3:38291)
0131 03:13:33.529032 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:33.529070 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260131 03:13:34.482766 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:34.482460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.2:47721)
0131 03:13:34.482620 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:34.482675 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
I20260131 03:13:34.529821 4501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:34.529572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.3:33307)
0131 03:13:34.529724 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:34.529770 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":62,"threads_started":1}
I20260131 03:13:34.529822 4502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:34.529581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.1:48411)
0131 03:13:34.529745 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:34.529778 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":103,"threads_started":1}
I20260131 03:13:35.483552 4503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:35.483292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.2:39445)
0131 03:13:35.483460 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:35.483501 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":69,"threads_started":1}
I20260131 03:13:35.530505 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:35.530256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.3:36709)
0131 03:13:35.530399 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:35.530447 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260131 03:13:35.530539 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:35.530349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:44007 (local address 127.19.84.1:54509)
0131 03:13:35.530473 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:35.530495 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.62:44007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
I20260131 03:13:35.939826 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 4008
I20260131 03:13:35.944999 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 4176
I20260131 03:13:35.949579 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 4312
2026-01-31T03:13:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59288 ms)
[----------] 31 tests from SecurityITest (366648 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:13:36 dist-test-slave-5nnp krb5kdc[4508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:13:36 dist-test-slave-5nnp krb5kdc[4508](info): set up 2 sockets
Jan 31 03:13:36 dist-test-slave-5nnp krb5kdc[4508](info): commencing operation
krb5kdc: starting...
W20260131 03:13:37.955713 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.985s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:13:38 dist-test-slave-5nnp krb5kdc[4508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:13:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:13:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:13:38.097904 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:42731
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:42731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:13:38.177058 4524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:38.177281 4524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:38.177320 4524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:38.178793 4524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:13:38.178864 4524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:38.178889 4524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:38.178910 4524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:13:38.178939 4524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:13:38.180538 4524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:42731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:42731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:38.180809 4524 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:38.180999 4524 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:38.183729 4532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:38.183725 4530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:38.183874 4524 server_base.cc:1060] running on GCE node
W20260131 03:13:38.183833 4529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:38.184335 4524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:38.184541 4524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:38.185709 4524 hybrid_clock.cc:648] HybridClock initialized: now 1769829218185681 us; error 40 us; skew 500 ppm
Jan 31 03:13:38 dist-test-slave-5nnp krb5kdc[4508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829218, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:13:38.188024 4524 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:13:38.188571 4524 webserver.cc:492] Webserver started at http://127.19.84.62:46775/ using document root <none> and password file <none>
I20260131 03:13:38.188732 4524 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:38.188766 4524 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:38.188854 4524 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:38.189759 4524 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "bacc5c92973b465ea08ef305aeaab11b"
format_stamp: "Formatted at 2026-01-31 03:13:38 on dist-test-slave-5nnp"
server_key: "a8797f8257a3742923d4085504af03d5"
server_key_iv: "b33a4c79c36d07531080c2783b518b40"
server_key_version: "encryptionkey@0"
I20260131 03:13:38.190032 4524 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "bacc5c92973b465ea08ef305aeaab11b"
format_stamp: "Formatted at 2026-01-31 03:13:38 on dist-test-slave-5nnp"
server_key: "a8797f8257a3742923d4085504af03d5"
server_key_iv: "b33a4c79c36d07531080c2783b518b40"
server_key_version: "encryptionkey@0"
I20260131 03:13:38.191169 4524 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260131 03:13:38.191939 4539 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:38.192178 4524 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260131 03:13:38.192242 4524 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "bacc5c92973b465ea08ef305aeaab11b"
format_stamp: "Formatted at 2026-01-31 03:13:38 on dist-test-slave-5nnp"
server_key: "a8797f8257a3742923d4085504af03d5"
server_key_iv: "b33a4c79c36d07531080c2783b518b40"
server_key_version: "encryptionkey@0"
I20260131 03:13:38.192328 4524 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:38.218955 4524 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:38.223860 4524 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:38.224013 4524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:38.228363 4524 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:42731
I20260131 03:13:38.228422 4591 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:42731 every 8 connection(s)
I20260131 03:13:38.228739 4524 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:13:38.229368 4592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:38.231855 4592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b: Bootstrap starting.
I20260131 03:13:38.232332 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 4524
I20260131 03:13:38.232457 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:13:38.232545 4592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:38.232640 19792 external_mini_cluster.cc:1468] Setting key 825355a87d895e0309fe227f2e8529ff
I20260131 03:13:38.232829 4592 log.cc:826] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:38.233696 4592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b: No bootstrap required, opened a new log
I20260131 03:13:38.235229 4592 raft_consensus.cc:359] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bacc5c92973b465ea08ef305aeaab11b" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42731 } }
I20260131 03:13:38.235370 4592 raft_consensus.cc:385] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:38.235427 4592 raft_consensus.cc:740] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bacc5c92973b465ea08ef305aeaab11b, State: Initialized, Role: FOLLOWER
I20260131 03:13:38.235538 4592 consensus_queue.cc:260] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bacc5c92973b465ea08ef305aeaab11b" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42731 } }
I20260131 03:13:38.235603 4592 raft_consensus.cc:399] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:13:38.235626 4592 raft_consensus.cc:493] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:13:38.235698 4592 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:38.236318 4592 raft_consensus.cc:515] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bacc5c92973b465ea08ef305aeaab11b" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42731 } }
I20260131 03:13:38.236452 4592 leader_election.cc:304] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bacc5c92973b465ea08ef305aeaab11b; no voters:
I20260131 03:13:38.236609 4592 leader_election.cc:290] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:13:38.236656 4597 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:13:38.236929 4592 sys_catalog.cc:565] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:13:38.236931 4597 raft_consensus.cc:697] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b [term 1 LEADER]: Becoming Leader. State: Replica: bacc5c92973b465ea08ef305aeaab11b, State: Running, Role: LEADER
I20260131 03:13:38.237128 4597 consensus_queue.cc:237] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bacc5c92973b465ea08ef305aeaab11b" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42731 } }
I20260131 03:13:38.237488 4598 sys_catalog.cc:455] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bacc5c92973b465ea08ef305aeaab11b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bacc5c92973b465ea08ef305aeaab11b" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42731 } } }
I20260131 03:13:38.237651 4598 sys_catalog.cc:458] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b [sys.catalog]: This master's current role is: LEADER
I20260131 03:13:38.237695 4599 sys_catalog.cc:455] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bacc5c92973b465ea08ef305aeaab11b. Latest consensus state: current_term: 1 leader_uuid: "bacc5c92973b465ea08ef305aeaab11b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bacc5c92973b465ea08ef305aeaab11b" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42731 } } }
I20260131 03:13:38.237857 4599 sys_catalog.cc:458] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b [sys.catalog]: This master's current role is: LEADER
Jan 31 03:13:38 dist-test-slave-5nnp krb5kdc[4508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
W20260131 03:13:38.239084 4613 catalog_manager.cc:1568] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:13:38.239144 4613 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:13:38.239210 4604 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:13:38.239421 4604 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:13:38.240886 4604 catalog_manager.cc:1357] Generated new cluster ID: 1bf806ee4c484ebebdbca46b93401aed
I20260131 03:13:38.240945 4604 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:13:38.243046 4596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:38.233172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54010 (local address 127.19.84.62:42731)
0131 03:13:38.233340 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:38.233352 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:38.233379 (+ 27us) server_negotiation.cc:408] Connection header received
0131 03:13:38.233734 (+ 355us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:38.233739 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:38.233810 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:38.233933 (+ 123us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:38.234938 (+ 1005us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.235859 (+ 921us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:38.236536 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.236826 (+ 290us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:38.239473 (+ 2647us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:38.239481 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:38.239482 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:38.239501 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:38.241599 (+ 2098us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:38.241990 (+ 391us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:38.241992 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:38.241993 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:38.242056 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:38.242235 (+ 179us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:38.242235 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:38.242236 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:38.242450 (+ 214us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:38.242584 (+ 134us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:38.242739 (+ 155us) server_negotiation.cc:300] Negotiation successful
0131 03:13:38.242852 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260131 03:13:38.248718 4604 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:13:38.249444 4604 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:13:38.259063 4604 catalog_manager.cc:6027] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b: Generated new TSK 0
I20260131 03:13:38.259284 4604 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:13:38.310865 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:42731
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:13:38.390843 4620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:38.391023 4620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:38.391047 4620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:38.392482 4620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:38.392525 4620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:38.392556 4620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:13:38.394101 4620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:42731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:38.394273 4620 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:38.394418 4620 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:38.397753 4628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:38.397791 4625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:38.397746 4626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:38.398041 4620 server_base.cc:1060] running on GCE node
I20260131 03:13:38.398202 4620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:38.398372 4620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:38.399500 4620 hybrid_clock.cc:648] HybridClock initialized: now 1769829218399489 us; error 26 us; skew 500 ppm
Jan 31 03:13:38 dist-test-slave-5nnp krb5kdc[4508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829218, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:13:38.401814 4620 init.cc:377] Logged in from keytab as kudu/127.19.84.1@KRBTEST.COM (short username kudu)
I20260131 03:13:38.402372 4620 webserver.cc:492] Webserver started at http://127.19.84.1:40347/ using document root <none> and password file <none>
I20260131 03:13:38.402573 4620 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:38.402629 4620 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:38.402791 4620 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:38.403827 4620 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "8b9ff50fafdd448eb7305e5236584934"
format_stamp: "Formatted at 2026-01-31 03:13:38 on dist-test-slave-5nnp"
server_key: "1035251a01eac2df2371a41438cda787"
server_key_iv: "52a5d36b9dc71b7a690aac3ae368aca8"
server_key_version: "encryptionkey@0"
I20260131 03:13:38.404183 4620 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "8b9ff50fafdd448eb7305e5236584934"
format_stamp: "Formatted at 2026-01-31 03:13:38 on dist-test-slave-5nnp"
server_key: "1035251a01eac2df2371a41438cda787"
server_key_iv: "52a5d36b9dc71b7a690aac3ae368aca8"
server_key_version: "encryptionkey@0"
I20260131 03:13:38.405465 4620 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:13:38.406243 4635 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:38.406424 4620 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260131 03:13:38.406514 4620 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "8b9ff50fafdd448eb7305e5236584934"
format_stamp: "Formatted at 2026-01-31 03:13:38 on dist-test-slave-5nnp"
server_key: "1035251a01eac2df2371a41438cda787"
server_key_iv: "52a5d36b9dc71b7a690aac3ae368aca8"
server_key_version: "encryptionkey@0"
I20260131 03:13:38.406598 4620 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:38.437690 4620 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:38.440465 4620 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:38.440627 4620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:38.440873 4620 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:13:38.441211 4620 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:13:38.441267 4620 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:38.441308 4620 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:13:38.441350 4620 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:38.447662 4620 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:40099
I20260131 03:13:38.447750 4748 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:40099 every 8 connection(s)
I20260131 03:13:38.448055 4620 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
Jan 31 03:13:38 dist-test-slave-5nnp krb5kdc[4508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829218, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:13:38.455978 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 4620
I20260131 03:13:38.456120 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:13:38.456358 19792 external_mini_cluster.cc:1468] Setting key 3a1f0f302bc0e8f5095b8e3e12e78dad
I20260131 03:13:38.456946 4596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:38.448784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:51937 (local address 127.19.84.62:42731)
0131 03:13:38.448836 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:38.448839 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:38.448999 (+ 160us) server_negotiation.cc:408] Connection header received
0131 03:13:38.449477 (+ 478us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:38.449478 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:38.449487 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:38.449547 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:38.450340 (+ 793us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.450896 (+ 556us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:38.451512 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.451654 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:38.453987 (+ 2333us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:38.453992 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:38.453992 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:38.453999 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:38.455469 (+ 1470us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:38.455883 (+ 414us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:38.455884 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:38.455884 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:38.455905 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:38.456184 (+ 279us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:38.456185 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:38.456186 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:38.456287 (+ 101us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:38.456343 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:38.456727 (+ 384us) server_negotiation.cc:300] Negotiation successful
0131 03:13:38.456832 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260131 03:13:38.457080 4751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:38.448753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:42731 (local address 127.19.84.1:51937)
0131 03:13:38.448900 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:38.448932 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:38.449395 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:38.449605 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:38.449609 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:38.449936 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:13:38.450226 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:38.450230 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.450984 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:38.450985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:38.451429 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:38.451435 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.451543 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:38.452495 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:13:38.452514 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:13:38.453905 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:13:38.455543 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:38.455549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:38.455552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:38.455795 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:38.455971 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:38.455972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:38.455972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:38.456079 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:38.456353 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:13:38.456359 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:13:38.456526 (+ 167us) client_negotiation.cc:769] Sending connection context
0131 03:13:38.456680 (+ 154us) client_negotiation.cc:240] Negotiation successful
0131 03:13:38.456846 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260131 03:13:38.457769 4749 heartbeater.cc:344] Connected to a master server at 127.19.84.62:42731
I20260131 03:13:38.457870 4749 heartbeater.cc:461] Registering TS with master...
I20260131 03:13:38.458103 4749 heartbeater.cc:507] Master 127.19.84.62:42731 requested a full tablet report, sending...
I20260131 03:13:38.458632 4556 ts_manager.cc:194] Registered new tserver with Master: 8b9ff50fafdd448eb7305e5236584934 (127.19.84.1:40099)
I20260131 03:13:38.459331 4556 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.1@KRBTEST.COM'} at 127.19.84.1:51937
WARNING: no policy specified for kudu/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:13:38.503077 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:42731
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:13:38.583415 4756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:38.583600 4756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:38.583624 4756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:38.585116 4756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:38.585165 4756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:38.585192 4756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:13:38.586778 4756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:42731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:38.587005 4756 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:38.587193 4756 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:38.589979 4762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:38.589985 4761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:38.590211 4764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:38.590198 4756 server_base.cc:1060] running on GCE node
I20260131 03:13:38.590400 4756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:38.590572 4756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:38.591701 4756 hybrid_clock.cc:648] HybridClock initialized: now 1769829218591689 us; error 26 us; skew 500 ppm
Jan 31 03:13:38 dist-test-slave-5nnp krb5kdc[4508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829218, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:13:38.594116 4756 init.cc:377] Logged in from keytab as kudu/127.19.84.2@KRBTEST.COM (short username kudu)
I20260131 03:13:38.594676 4756 webserver.cc:492] Webserver started at http://127.19.84.2:34995/ using document root <none> and password file <none>
I20260131 03:13:38.594888 4756 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:38.594929 4756 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:38.595057 4756 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:38.596114 4756 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "f91d60153499403d8f7ff9116d357c15"
format_stamp: "Formatted at 2026-01-31 03:13:38 on dist-test-slave-5nnp"
server_key: "2ac7b76fa6236b8c8ae936c09b2cbeec"
server_key_iv: "d4030e14688bd733fecb7d9f7ed04b0b"
server_key_version: "encryptionkey@0"
I20260131 03:13:38.596494 4756 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "f91d60153499403d8f7ff9116d357c15"
format_stamp: "Formatted at 2026-01-31 03:13:38 on dist-test-slave-5nnp"
server_key: "2ac7b76fa6236b8c8ae936c09b2cbeec"
server_key_iv: "d4030e14688bd733fecb7d9f7ed04b0b"
server_key_version: "encryptionkey@0"
I20260131 03:13:38.597747 4756 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:13:38.598506 4771 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:38.598891 4756 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:13:38.599092 4756 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "f91d60153499403d8f7ff9116d357c15"
format_stamp: "Formatted at 2026-01-31 03:13:38 on dist-test-slave-5nnp"
server_key: "2ac7b76fa6236b8c8ae936c09b2cbeec"
server_key_iv: "d4030e14688bd733fecb7d9f7ed04b0b"
server_key_version: "encryptionkey@0"
I20260131 03:13:38.599176 4756 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:38.611960 4756 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:38.613888 4756 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:38.614063 4756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:38.614300 4756 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:13:38.614629 4756 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:13:38.614683 4756 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:38.614745 4756 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:13:38.614784 4756 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:38.620800 4756 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:40883
I20260131 03:13:38.620859 4884 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:40883 every 8 connection(s)
I20260131 03:13:38.621181 4756 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
Jan 31 03:13:38 dist-test-slave-5nnp krb5kdc[4508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829218, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:13:38.627686 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 4756
I20260131 03:13:38.628024 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:13:38.628242 19792 external_mini_cluster.cc:1468] Setting key 00ed9d458c0941a6a0c31ceab10694c6
I20260131 03:13:38.629920 4596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:38.621907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:36125 (local address 127.19.84.62:42731)
0131 03:13:38.621988 (+ 81us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:38.621990 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:38.622085 (+ 95us) server_negotiation.cc:408] Connection header received
0131 03:13:38.622581 (+ 496us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:38.622583 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:38.622594 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:38.622652 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:38.623490 (+ 838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.623987 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:38.624557 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.624666 (+ 109us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:38.626994 (+ 2328us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:38.626999 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:38.626999 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:38.627005 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:38.628560 (+ 1555us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:38.629013 (+ 453us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:38.629014 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:38.629015 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:38.629052 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:38.629281 (+ 229us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:38.629282 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:38.629282 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:38.629385 (+ 103us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:38.629457 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:38.629738 (+ 281us) server_negotiation.cc:300] Negotiation successful
0131 03:13:38.629831 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260131 03:13:38.630046 4887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:38.621842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:42731 (local address 127.19.84.2:36125)
0131 03:13:38.621975 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:38.622008 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:38.622465 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:38.622721 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:38.622725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:38.623050 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:13:38.623330 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:38.623333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.624083 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:38.624084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:38.624485 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:38.624487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.624596 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:38.625517 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:13:38.625534 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:13:38.626910 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:13:38.628674 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:38.628680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:38.628683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:38.628920 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:38.629116 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:38.629117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:38.629118 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:38.629200 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:38.629449 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:13:38.629454 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:13:38.629584 (+ 130us) client_negotiation.cc:769] Sending connection context
0131 03:13:38.629708 (+ 124us) client_negotiation.cc:240] Negotiation successful
0131 03:13:38.629854 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":57,"threads_started":1}
I20260131 03:13:38.630487 4885 heartbeater.cc:344] Connected to a master server at 127.19.84.62:42731
I20260131 03:13:38.630602 4885 heartbeater.cc:461] Registering TS with master...
I20260131 03:13:38.630843 4885 heartbeater.cc:507] Master 127.19.84.62:42731 requested a full tablet report, sending...
I20260131 03:13:38.631247 4556 ts_manager.cc:194] Registered new tserver with Master: f91d60153499403d8f7ff9116d357c15 (127.19.84.2:40883)
I20260131 03:13:38.631677 4556 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.2@KRBTEST.COM'} at 127.19.84.2:36125
WARNING: no policy specified for kudu/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:13:38.677749 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:42731
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:13:38.758312 4892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:38.758513 4892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:38.758538 4892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:38.760603 4892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:38.760665 4892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:38.760696 4892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:13:38.762492 4892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:42731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:38.762733 4892 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:38.762959 4892 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:38.765805 4898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:38.765827 4897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:38.765870 4892 server_base.cc:1060] running on GCE node
W20260131 03:13:38.765864 4900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:38.766301 4892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:38.766480 4892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:38.767632 4892 hybrid_clock.cc:648] HybridClock initialized: now 1769829218767605 us; error 40 us; skew 500 ppm
Jan 31 03:13:38 dist-test-slave-5nnp krb5kdc[4508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829218, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:13:38.770512 4892 init.cc:377] Logged in from keytab as kudu/127.19.84.3@KRBTEST.COM (short username kudu)
I20260131 03:13:38.771162 4892 webserver.cc:492] Webserver started at http://127.19.84.3:43803/ using document root <none> and password file <none>
I20260131 03:13:38.771412 4892 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:38.771481 4892 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:38.771605 4892 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:38.772614 4892 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "2de4723058ba4477a496b50ddfeb1c67"
format_stamp: "Formatted at 2026-01-31 03:13:38 on dist-test-slave-5nnp"
server_key: "b14a81e995034519c925381a93609379"
server_key_iv: "785d2c7fc966edcb01c2180a58432fe6"
server_key_version: "encryptionkey@0"
I20260131 03:13:38.772946 4892 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "2de4723058ba4477a496b50ddfeb1c67"
format_stamp: "Formatted at 2026-01-31 03:13:38 on dist-test-slave-5nnp"
server_key: "b14a81e995034519c925381a93609379"
server_key_iv: "785d2c7fc966edcb01c2180a58432fe6"
server_key_version: "encryptionkey@0"
I20260131 03:13:38.774231 4892 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:13:38.774959 4907 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:38.775184 4892 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:13:38.775274 4892 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "2de4723058ba4477a496b50ddfeb1c67"
format_stamp: "Formatted at 2026-01-31 03:13:38 on dist-test-slave-5nnp"
server_key: "b14a81e995034519c925381a93609379"
server_key_iv: "785d2c7fc966edcb01c2180a58432fe6"
server_key_version: "encryptionkey@0"
I20260131 03:13:38.775355 4892 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:38.788219 4892 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:38.790387 4892 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:38.790498 4892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:38.790669 4892 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:13:38.790940 4892 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:13:38.790970 4892 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:38.790994 4892 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:13:38.791006 4892 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:38.796936 4892 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:42775
I20260131 03:13:38.797015 5020 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:42775 every 8 connection(s)
I20260131 03:13:38.797322 4892 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
I20260131 03:13:38.802312 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 4892
I20260131 03:13:38.802407 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:13:38.802603 19792 external_mini_cluster.cc:1468] Setting key 9b60abc3bf296f33e30f1230b94ab953
Jan 31 03:13:38 dist-test-slave-5nnp krb5kdc[4508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829218, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:13:38.806628 4596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:38.798030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:39093 (local address 127.19.84.62:42731)
0131 03:13:38.798137 (+ 107us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:38.798139 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:38.798289 (+ 150us) server_negotiation.cc:408] Connection header received
0131 03:13:38.798832 (+ 543us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:38.798833 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:38.798847 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:38.798909 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:38.799789 (+ 880us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.800293 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:38.800889 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.801077 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:38.803719 (+ 2642us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:38.803722 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:38.803723 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:38.803730 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:38.805083 (+ 1353us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:38.805658 (+ 575us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:38.805659 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:38.805660 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:38.805695 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:38.805954 (+ 259us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:38.805954 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:38.805955 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:38.806078 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:38.806159 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:38.806434 (+ 275us) server_negotiation.cc:300] Negotiation successful
0131 03:13:38.806537 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260131 03:13:38.806757 5023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:38.797959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:42731 (local address 127.19.84.3:39093)
0131 03:13:38.798176 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:38.798205 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:38.798703 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:38.799003 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:38.799007 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:38.799320 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:13:38.799676 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:38.799680 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.800382 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:38.800384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:38.800771 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:38.800772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.800954 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:38.801899 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:13:38.801915 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:13:38.803626 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:13:38.805203 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:38.805210 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:38.805213 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:38.805545 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:38.805805 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:38.805806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:38.805806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:38.805882 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:38.806147 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:13:38.806152 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:13:38.806290 (+ 138us) client_negotiation.cc:769] Sending connection context
0131 03:13:38.806417 (+ 127us) client_negotiation.cc:240] Negotiation successful
0131 03:13:38.806558 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":76,"threads_started":1}
I20260131 03:13:38.807278 5021 heartbeater.cc:344] Connected to a master server at 127.19.84.62:42731
I20260131 03:13:38.807417 5021 heartbeater.cc:461] Registering TS with master...
I20260131 03:13:38.807658 5021 heartbeater.cc:507] Master 127.19.84.62:42731 requested a full tablet report, sending...
I20260131 03:13:38.808075 4556 ts_manager.cc:194] Registered new tserver with Master: 2de4723058ba4477a496b50ddfeb1c67 (127.19.84.3:42775)
I20260131 03:13:38.808455 4556 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:39093
I20260131 03:13:38.814756 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:13:38.822707 4596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:38.816299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54022 (local address 127.19.84.62:42731)
0131 03:13:38.816363 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:38.816366 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:38.816498 (+ 132us) server_negotiation.cc:408] Connection header received
0131 03:13:38.816567 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:38.816568 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:38.816580 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:38.816646 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:38.817632 (+ 986us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.818372 (+ 740us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:38.819189 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.819364 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:38.820419 (+ 1055us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:38.820422 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:38.820423 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:38.820429 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:38.821782 (+ 1353us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:38.822111 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:38.822112 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:38.822113 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:38.822135 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:38.822271 (+ 136us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:38.822272 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:38.822272 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:38.822387 (+ 115us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:38.822466 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:38.822567 (+ 101us) server_negotiation.cc:300] Negotiation successful
0131 03:13:38.822637 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260131 03:13:38.824640 4556 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:13:38.825980 4556 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:13:38.831736 5032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:38.828421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:40099 (local address 127.0.0.1:39252)
0131 03:13:38.828652 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:38.828665 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:38.828733 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:38.829449 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:38.829450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:38.829452 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:38.829699 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:38.829701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.830595 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:38.830597 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:38.831472 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:38.831474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.831542 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:38.831579 (+ 37us) client_negotiation.cc:769] Sending connection context
0131 03:13:38.831623 (+ 44us) client_negotiation.cc:240] Negotiation successful
0131 03:13:38.831659 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":113,"threads_started":1}
I20260131 03:13:38.831988 5036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:38.828614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:42775 (local address 127.0.0.1:53096)
0131 03:13:38.828752 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:38.828762 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:38.828810 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:38.829544 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:38.829545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:38.829547 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:38.829732 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:38.829735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.830981 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:38.830982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:38.831754 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:38.831756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.831845 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:38.831849 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:13:38.831881 (+ 32us) client_negotiation.cc:240] Negotiation successful
0131 03:13:38.831916 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260131 03:13:38.832355 5033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:38.828416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:40883 (local address 127.0.0.1:33104)
0131 03:13:38.828607 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:38.828643 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:38.828730 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:38.829143 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:38.829146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:38.829148 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:38.829455 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:38.829458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.830901 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:38.830901 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:38.831745 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:38.831747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.832215 (+ 468us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:38.832218 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:13:38.832253 (+ 35us) client_negotiation.cc:240] Negotiation successful
0131 03:13:38.832293 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":59,"threads_started":1}
I20260131 03:13:38.832489 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:38.828829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53096 (local address 127.19.84.3:42775)
0131 03:13:38.828987 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:38.828997 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:38.829005 (+ 8us) server_negotiation.cc:408] Connection header received
0131 03:13:38.829034 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:38.829035 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:38.829052 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:38.829134 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:38.829818 (+ 684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.830914 (+ 1096us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:38.831857 (+ 943us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.832342 (+ 485us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:38.832382 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:38.832406 (+ 24us) server_negotiation.cc:300] Negotiation successful
0131 03:13:38.832432 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
I20260131 03:13:38.832758 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:38.828501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39252 (local address 127.19.84.1:40099)
0131 03:13:38.829247 (+ 746us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:38.829252 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:38.829260 (+ 8us) server_negotiation.cc:408] Connection header received
0131 03:13:38.829289 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:38.829290 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:38.829302 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:38.829384 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:38.829811 (+ 427us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.830511 (+ 700us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:38.831960 (+ 1449us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.832538 (+ 578us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:38.832600 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:38.832633 (+ 33us) server_negotiation.cc:300] Negotiation successful
0131 03:13:38.832678 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":81,"threads_started":1}
I20260131 03:13:38.833094 4955 tablet_service.cc:1505] Processing CreateTablet for tablet 384b36aaa72644278338460bed686dfa (DEFAULT_TABLE table=test-table [id=0acbe01a7a7944fbb980e1c4eefb0412]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:13:38.833333 4683 tablet_service.cc:1505] Processing CreateTablet for tablet 384b36aaa72644278338460bed686dfa (DEFAULT_TABLE table=test-table [id=0acbe01a7a7944fbb980e1c4eefb0412]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:13:38.833417 4955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 384b36aaa72644278338460bed686dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:38.833591 4683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 384b36aaa72644278338460bed686dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:38.836037 5035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:38.828499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33104 (local address 127.19.84.2:40883)
0131 03:13:38.828988 (+ 489us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:38.828995 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:38.829004 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:13:38.829038 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:38.829039 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:38.829052 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:38.829129 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:38.829530 (+ 401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.830819 (+ 1289us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:38.835495 (+ 4676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:38.835892 (+ 397us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:38.835928 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:38.835951 (+ 23us) server_negotiation.cc:300] Negotiation successful
0131 03:13:38.835981 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":90,"threads_started":1}
I20260131 03:13:38.836644 4819 tablet_service.cc:1505] Processing CreateTablet for tablet 384b36aaa72644278338460bed686dfa (DEFAULT_TABLE table=test-table [id=0acbe01a7a7944fbb980e1c4eefb0412]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:13:38.837006 4819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 384b36aaa72644278338460bed686dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:38.839303 5040 tablet_bootstrap.cc:492] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15: Bootstrap starting.
I20260131 03:13:38.839421 5039 tablet_bootstrap.cc:492] T 384b36aaa72644278338460bed686dfa P 8b9ff50fafdd448eb7305e5236584934: Bootstrap starting.
I20260131 03:13:38.839920 5040 tablet_bootstrap.cc:654] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:38.839969 5039 tablet_bootstrap.cc:654] T 384b36aaa72644278338460bed686dfa P 8b9ff50fafdd448eb7305e5236584934: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:38.840288 5039 log.cc:826] T 384b36aaa72644278338460bed686dfa P 8b9ff50fafdd448eb7305e5236584934: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:38.840566 5040 log.cc:826] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:38.840701 5038 tablet_bootstrap.cc:492] T 384b36aaa72644278338460bed686dfa P 2de4723058ba4477a496b50ddfeb1c67: Bootstrap starting.
I20260131 03:13:38.841236 5038 tablet_bootstrap.cc:654] T 384b36aaa72644278338460bed686dfa P 2de4723058ba4477a496b50ddfeb1c67: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:38.841414 5040 tablet_bootstrap.cc:492] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15: No bootstrap required, opened a new log
I20260131 03:13:38.841491 5040 ts_tablet_manager.cc:1403] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:13:38.841526 5038 log.cc:826] T 384b36aaa72644278338460bed686dfa P 2de4723058ba4477a496b50ddfeb1c67: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:38.842310 5038 tablet_bootstrap.cc:492] T 384b36aaa72644278338460bed686dfa P 2de4723058ba4477a496b50ddfeb1c67: No bootstrap required, opened a new log
I20260131 03:13:38.842396 5038 ts_tablet_manager.cc:1403] T 384b36aaa72644278338460bed686dfa P 2de4723058ba4477a496b50ddfeb1c67: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:13:38.843014 5040 raft_consensus.cc:359] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9ff50fafdd448eb7305e5236584934" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 40099 } } peers { permanent_uuid: "f91d60153499403d8f7ff9116d357c15" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40883 } } peers { permanent_uuid: "2de4723058ba4477a496b50ddfeb1c67" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 42775 } }
I20260131 03:13:38.843154 5040 raft_consensus.cc:385] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:38.843199 5040 raft_consensus.cc:740] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f91d60153499403d8f7ff9116d357c15, State: Initialized, Role: FOLLOWER
I20260131 03:13:38.843300 5040 consensus_queue.cc:260] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9ff50fafdd448eb7305e5236584934" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 40099 } } peers { permanent_uuid: "f91d60153499403d8f7ff9116d357c15" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40883 } } peers { permanent_uuid: "2de4723058ba4477a496b50ddfeb1c67" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 42775 } }
I20260131 03:13:38.843567 5040 ts_tablet_manager.cc:1434] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:13:38.843683 4885 heartbeater.cc:499] Master 127.19.84.62:42731 was elected leader, sending a full tablet report...
I20260131 03:13:38.843874 5038 raft_consensus.cc:359] T 384b36aaa72644278338460bed686dfa P 2de4723058ba4477a496b50ddfeb1c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9ff50fafdd448eb7305e5236584934" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 40099 } } peers { permanent_uuid: "f91d60153499403d8f7ff9116d357c15" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40883 } } peers { permanent_uuid: "2de4723058ba4477a496b50ddfeb1c67" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 42775 } }
I20260131 03:13:38.844028 5038 raft_consensus.cc:385] T 384b36aaa72644278338460bed686dfa P 2de4723058ba4477a496b50ddfeb1c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:38.844074 5038 raft_consensus.cc:740] T 384b36aaa72644278338460bed686dfa P 2de4723058ba4477a496b50ddfeb1c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2de4723058ba4477a496b50ddfeb1c67, State: Initialized, Role: FOLLOWER
I20260131 03:13:38.844165 5038 consensus_queue.cc:260] T 384b36aaa72644278338460bed686dfa P 2de4723058ba4477a496b50ddfeb1c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9ff50fafdd448eb7305e5236584934" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 40099 } } peers { permanent_uuid: "f91d60153499403d8f7ff9116d357c15" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40883 } } peers { permanent_uuid: "2de4723058ba4477a496b50ddfeb1c67" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 42775 } }
I20260131 03:13:38.844390 5038 ts_tablet_manager.cc:1434] T 384b36aaa72644278338460bed686dfa P 2de4723058ba4477a496b50ddfeb1c67: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260131 03:13:38.844455 5021 heartbeater.cc:499] Master 127.19.84.62:42731 was elected leader, sending a full tablet report...
I20260131 03:13:38.844805 5039 tablet_bootstrap.cc:492] T 384b36aaa72644278338460bed686dfa P 8b9ff50fafdd448eb7305e5236584934: No bootstrap required, opened a new log
I20260131 03:13:38.844877 5039 ts_tablet_manager.cc:1403] T 384b36aaa72644278338460bed686dfa P 8b9ff50fafdd448eb7305e5236584934: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260131 03:13:38.846274 5039 raft_consensus.cc:359] T 384b36aaa72644278338460bed686dfa P 8b9ff50fafdd448eb7305e5236584934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9ff50fafdd448eb7305e5236584934" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 40099 } } peers { permanent_uuid: "f91d60153499403d8f7ff9116d357c15" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40883 } } peers { permanent_uuid: "2de4723058ba4477a496b50ddfeb1c67" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 42775 } }
I20260131 03:13:38.846382 5039 raft_consensus.cc:385] T 384b36aaa72644278338460bed686dfa P 8b9ff50fafdd448eb7305e5236584934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:38.846400 5039 raft_consensus.cc:740] T 384b36aaa72644278338460bed686dfa P 8b9ff50fafdd448eb7305e5236584934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9ff50fafdd448eb7305e5236584934, State: Initialized, Role: FOLLOWER
I20260131 03:13:38.846491 5039 consensus_queue.cc:260] T 384b36aaa72644278338460bed686dfa P 8b9ff50fafdd448eb7305e5236584934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9ff50fafdd448eb7305e5236584934" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 40099 } } peers { permanent_uuid: "f91d60153499403d8f7ff9116d357c15" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40883 } } peers { permanent_uuid: "2de4723058ba4477a496b50ddfeb1c67" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 42775 } }
I20260131 03:13:38.846684 5039 ts_tablet_manager.cc:1434] T 384b36aaa72644278338460bed686dfa P 8b9ff50fafdd448eb7305e5236584934: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:13:38.846764 4749 heartbeater.cc:499] Master 127.19.84.62:42731 was elected leader, sending a full tablet report...
W20260131 03:13:38.871766 4886 tablet.cc:2378] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260131 03:13:38.948760 4750 tablet.cc:2378] T 384b36aaa72644278338460bed686dfa P 8b9ff50fafdd448eb7305e5236584934: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:13:39.017062 5044 raft_consensus.cc:493] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:13:39.017233 5044 raft_consensus.cc:515] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9ff50fafdd448eb7305e5236584934" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 40099 } } peers { permanent_uuid: "f91d60153499403d8f7ff9116d357c15" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40883 } } peers { permanent_uuid: "2de4723058ba4477a496b50ddfeb1c67" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 42775 } }
I20260131 03:13:39.017624 5044 leader_election.cc:290] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b9ff50fafdd448eb7305e5236584934 (127.19.84.1:40099), 2de4723058ba4477a496b50ddfeb1c67 (127.19.84.3:42775)
I20260131 03:13:39.019975 4887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.017849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:40099 (local address 127.19.84.2:55093)
0131 03:13:39.017943 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:39.017955 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:39.018044 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:39.018233 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:39.018234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:39.018236 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:39.018421 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:39.018423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.019079 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:39.019080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:39.019718 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:39.019720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.019819 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:39.019823 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:13:39.019857 (+ 34us) client_negotiation.cc:240] Negotiation successful
0131 03:13:39.019900 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260131 03:13:39.020298 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.017852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:42775 (local address 127.19.84.2:44827)
0131 03:13:39.018124 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:39.018134 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:39.018220 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:39.018397 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:39.018398 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:39.018399 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:39.018628 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:39.018629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.019319 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:39.019320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:39.020014 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:39.020017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.020120 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:39.020125 (+ 5us) client_negotiation.cc:769] Sending connection context
0131 03:13:39.020162 (+ 37us) client_negotiation.cc:240] Negotiation successful
0131 03:13:39.020202 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":40,"thread_start_us":158,"threads_started":1}
I20260131 03:13:39.020474 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.017880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:55093 (local address 127.19.84.1:40099)
0131 03:13:39.017932 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:39.017935 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:39.018030 (+ 95us) server_negotiation.cc:408] Connection header received
0131 03:13:39.018136 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:39.018138 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:39.018151 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:39.018220 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:39.018498 (+ 278us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.018985 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:39.019816 (+ 831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.020297 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:39.020307 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:39.020338 (+ 31us) server_negotiation.cc:300] Negotiation successful
0131 03:13:39.020392 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260131 03:13:39.020874 4703 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "384b36aaa72644278338460bed686dfa" candidate_uuid: "f91d60153499403d8f7ff9116d357c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9ff50fafdd448eb7305e5236584934" is_pre_election: true
I20260131 03:13:39.020920 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.017935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:44827 (local address 127.19.84.3:42775)
0131 03:13:39.017977 (+ 42us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:39.017979 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:39.018206 (+ 227us) server_negotiation.cc:408] Connection header received
0131 03:13:39.018297 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:39.018298 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:39.018311 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:39.018393 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:39.018709 (+ 316us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.019199 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:39.020205 (+ 1006us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.020808 (+ 603us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:39.020815 (+ 7us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:39.020835 (+ 20us) server_negotiation.cc:300] Negotiation successful
0131 03:13:39.020872 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260131 03:13:39.021054 4703 raft_consensus.cc:2468] T 384b36aaa72644278338460bed686dfa P 8b9ff50fafdd448eb7305e5236584934 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f91d60153499403d8f7ff9116d357c15 in term 0.
I20260131 03:13:39.021359 4774 leader_election.cc:304] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b9ff50fafdd448eb7305e5236584934, f91d60153499403d8f7ff9116d357c15; no voters:
I20260131 03:13:39.021435 4975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "384b36aaa72644278338460bed686dfa" candidate_uuid: "f91d60153499403d8f7ff9116d357c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de4723058ba4477a496b50ddfeb1c67" is_pre_election: true
I20260131 03:13:39.021481 5044 raft_consensus.cc:2804] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:13:39.021543 5044 raft_consensus.cc:493] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:13:39.021587 5044 raft_consensus.cc:3060] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:39.021593 4975 raft_consensus.cc:2468] T 384b36aaa72644278338460bed686dfa P 2de4723058ba4477a496b50ddfeb1c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f91d60153499403d8f7ff9116d357c15 in term 0.
I20260131 03:13:39.022300 5044 raft_consensus.cc:515] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9ff50fafdd448eb7305e5236584934" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 40099 } } peers { permanent_uuid: "f91d60153499403d8f7ff9116d357c15" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40883 } } peers { permanent_uuid: "2de4723058ba4477a496b50ddfeb1c67" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 42775 } }
I20260131 03:13:39.022522 5044 leader_election.cc:290] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15 [CANDIDATE]: Term 1 election: Requested vote from peers 8b9ff50fafdd448eb7305e5236584934 (127.19.84.1:40099), 2de4723058ba4477a496b50ddfeb1c67 (127.19.84.3:42775)
I20260131 03:13:39.022668 4703 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "384b36aaa72644278338460bed686dfa" candidate_uuid: "f91d60153499403d8f7ff9116d357c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9ff50fafdd448eb7305e5236584934"
I20260131 03:13:39.022750 4703 raft_consensus.cc:3060] T 384b36aaa72644278338460bed686dfa P 8b9ff50fafdd448eb7305e5236584934 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:39.022724 4975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "384b36aaa72644278338460bed686dfa" candidate_uuid: "f91d60153499403d8f7ff9116d357c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2de4723058ba4477a496b50ddfeb1c67"
I20260131 03:13:39.022845 4975 raft_consensus.cc:3060] T 384b36aaa72644278338460bed686dfa P 2de4723058ba4477a496b50ddfeb1c67 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:39.023635 4703 raft_consensus.cc:2468] T 384b36aaa72644278338460bed686dfa P 8b9ff50fafdd448eb7305e5236584934 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f91d60153499403d8f7ff9116d357c15 in term 1.
I20260131 03:13:39.023650 4975 raft_consensus.cc:2468] T 384b36aaa72644278338460bed686dfa P 2de4723058ba4477a496b50ddfeb1c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f91d60153499403d8f7ff9116d357c15 in term 1.
I20260131 03:13:39.023842 4773 leader_election.cc:304] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2de4723058ba4477a496b50ddfeb1c67, f91d60153499403d8f7ff9116d357c15; no voters:
I20260131 03:13:39.023988 5044 raft_consensus.cc:2804] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:13:39.024183 5044 raft_consensus.cc:697] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15 [term 1 LEADER]: Becoming Leader. State: Replica: f91d60153499403d8f7ff9116d357c15, State: Running, Role: LEADER
I20260131 03:13:39.024273 5044 consensus_queue.cc:237] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9ff50fafdd448eb7305e5236584934" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 40099 } } peers { permanent_uuid: "f91d60153499403d8f7ff9116d357c15" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40883 } } peers { permanent_uuid: "2de4723058ba4477a496b50ddfeb1c67" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 42775 } }
I20260131 03:13:39.025190 4556 catalog_manager.cc:5654] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15 reported cstate change: term changed from 0 to 1, leader changed from <none> to f91d60153499403d8f7ff9116d357c15 (127.19.84.2). New cstate: current_term: 1 leader_uuid: "f91d60153499403d8f7ff9116d357c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9ff50fafdd448eb7305e5236584934" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 40099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f91d60153499403d8f7ff9116d357c15" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 40883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2de4723058ba4477a496b50ddfeb1c67" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 42775 } health_report { overall_health: UNKNOWN } } }
W20260131 03:13:39.047915 5022 tablet.cc:2378] T 384b36aaa72644278338460bed686dfa P 2de4723058ba4477a496b50ddfeb1c67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:13:39.060232 5035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.058039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33120 (local address 127.19.84.2:40883)
0131 03:13:39.058134 (+ 95us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:39.058136 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:39.058144 (+ 8us) server_negotiation.cc:408] Connection header received
0131 03:13:39.058187 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:39.058187 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:39.058209 (+ 22us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:39.058262 (+ 53us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:13:39.058514 (+ 252us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.058993 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:39.059642 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.059787 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:39.059824 (+ 37us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:13:39.059912 (+ 88us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:13:39.059976 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:39.060085 (+ 109us) server_negotiation.cc:300] Negotiation successful
0131 03:13:39.060115 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260131 03:13:39.062152 4703 raft_consensus.cc:1275] T 384b36aaa72644278338460bed686dfa P 8b9ff50fafdd448eb7305e5236584934 [term 1 FOLLOWER]: Refusing update from remote peer f91d60153499403d8f7ff9116d357c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:13:39.062198 4975 raft_consensus.cc:1275] T 384b36aaa72644278338460bed686dfa P 2de4723058ba4477a496b50ddfeb1c67 [term 1 FOLLOWER]: Refusing update from remote peer f91d60153499403d8f7ff9116d357c15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:13:39.062515 5044 consensus_queue.cc:1048] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2de4723058ba4477a496b50ddfeb1c67" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 42775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:13:39.062670 5048 consensus_queue.cc:1048] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9ff50fafdd448eb7305e5236584934" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 40099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:13:39.065235 5051 mvcc.cc:204] Tried to move back new op lower bound from 7249220481275240448 to 7249220481124003840. Current Snapshot: MvccSnapshot[applied={T|T < 7249220481275240448}]
I20260131 03:13:39.065762 5052 mvcc.cc:204] Tried to move back new op lower bound from 7249220481275240448 to 7249220481124003840. Current Snapshot: MvccSnapshot[applied={T|T < 7249220481275240448}]
I20260131 03:13:39.065832 5053 mvcc.cc:204] Tried to move back new op lower bound from 7249220481275240448 to 7249220481124003840. Current Snapshot: MvccSnapshot[applied={T|T < 7249220481275240448}]
I20260131 03:13:39.067481 4556 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:13:39.067580 4556 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:13:39.068325 4556 catalog_manager.cc:5941] T 00000000000000000000000000000000 P bacc5c92973b465ea08ef305aeaab11b: Sending DeleteTablet for 3 replicas of tablet 384b36aaa72644278338460bed686dfa
I20260131 03:13:39.068689 4819 tablet_service.cc:1552] Processing DeleteTablet for tablet 384b36aaa72644278338460bed686dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:13:39 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:33104
I20260131 03:13:39.068706 4683 tablet_service.cc:1552] Processing DeleteTablet for tablet 384b36aaa72644278338460bed686dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:13:39 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:39252
I20260131 03:13:39.068703 4955 tablet_service.cc:1552] Processing DeleteTablet for tablet 384b36aaa72644278338460bed686dfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:13:39 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:53096
I20260131 03:13:39.068912 5062 tablet_replica.cc:333] T 384b36aaa72644278338460bed686dfa P 2de4723058ba4477a496b50ddfeb1c67: stopping tablet replica
I20260131 03:13:39.069002 5062 raft_consensus.cc:2243] T 384b36aaa72644278338460bed686dfa P 2de4723058ba4477a496b50ddfeb1c67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:13:39.069042 5060 tablet_replica.cc:333] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15: stopping tablet replica
I20260131 03:13:39.069139 5060 raft_consensus.cc:2243] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:13:39.069165 5062 raft_consensus.cc:2272] T 384b36aaa72644278338460bed686dfa P 2de4723058ba4477a496b50ddfeb1c67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:13:39.069258 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 4620
I20260131 03:13:39.069267 5060 raft_consensus.cc:2272] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:13:39.069615 5062 ts_tablet_manager.cc:1916] T 384b36aaa72644278338460bed686dfa P 2de4723058ba4477a496b50ddfeb1c67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:13:39.070345 5060 ts_tablet_manager.cc:1916] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:13:39.071391 5062 ts_tablet_manager.cc:1929] T 384b36aaa72644278338460bed686dfa P 2de4723058ba4477a496b50ddfeb1c67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:13:39.071456 5062 log.cc:1199] T 384b36aaa72644278338460bed686dfa P 2de4723058ba4477a496b50ddfeb1c67: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/384b36aaa72644278338460bed686dfa
I20260131 03:13:39.071731 5062 ts_tablet_manager.cc:1950] T 384b36aaa72644278338460bed686dfa P 2de4723058ba4477a496b50ddfeb1c67: Deleting consensus metadata
I20260131 03:13:39.072103 5060 ts_tablet_manager.cc:1929] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:13:39.072161 5060 log.cc:1199] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1769828849314684-19792-0/minicluster-data/ts-1/wal/wals/384b36aaa72644278338460bed686dfa
I20260131 03:13:39.072152 4541 catalog_manager.cc:4985] TS 2de4723058ba4477a496b50ddfeb1c67 (127.19.84.3:42775): tablet 384b36aaa72644278338460bed686dfa (table test-table [id=0acbe01a7a7944fbb980e1c4eefb0412]) successfully deleted
I20260131 03:13:39.072415 5060 ts_tablet_manager.cc:1950] T 384b36aaa72644278338460bed686dfa P f91d60153499403d8f7ff9116d357c15: Deleting consensus metadata
I20260131 03:13:39.072705 4541 catalog_manager.cc:4985] TS f91d60153499403d8f7ff9116d357c15 (127.19.84.2:40883): tablet 384b36aaa72644278338460bed686dfa (table test-table [id=0acbe01a7a7944fbb980e1c4eefb0412]) successfully deleted
W20260131 03:13:39.077395 4542 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.84.1:40099 (error 108)
I20260131 03:13:39.077656 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 4756
I20260131 03:13:39.078382 5032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.077820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:40099 (local address 127.0.0.1:39262)
0131 03:13:39.078315 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:39.078354 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.1:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474}
W20260131 03:13:39.078519 4542 catalog_manager.cc:4712] TS 8b9ff50fafdd448eb7305e5236584934 (127.19.84.1:40099): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 384b36aaa72644278338460bed686dfa: Network error: Client connection negotiation failed: client connection to 127.19.84.1:40099: connect: Connection refused (error 111)
I20260131 03:13:39.083272 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 4892
I20260131 03:13:39.087985 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 4524
2026-01-31T03:13:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (3137 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-01-31T03:13:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:13:39Z Disabled control of system clock
I20260131 03:13:39.115583 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:40155
--webserver_interface=127.19.84.62
--webserver_port=0
--builtin_ntp_servers=127.19.84.20:42215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:40155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260131 03:13:39.194954 5067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:39.195118 5067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:39.195137 5067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:39.196806 5067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:13:39.196851 5067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:39.196869 5067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:39.196882 5067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:13:39.196914 5067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:13:39.198437 5067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:42215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:40155
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:40155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:39.198671 5067 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:39.198911 5067 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:39.201488 5072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:39.201602 5067 server_base.cc:1060] running on GCE node
W20260131 03:13:39.201586 5075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:39.201478 5073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:39.201941 5067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:39.202176 5067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:39.203332 5067 hybrid_clock.cc:648] HybridClock initialized: now 1769829219203331 us; error 59 us; skew 500 ppm
I20260131 03:13:39.204535 5067 webserver.cc:492] Webserver started at http://127.19.84.62:38909/ using document root <none> and password file <none>
I20260131 03:13:39.204768 5067 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:39.204830 5067 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:39.204999 5067 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:39.205960 5067 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "3a25b27758f042edad8e61f3fa29147c"
format_stamp: "Formatted at 2026-01-31 03:13:39 on dist-test-slave-5nnp"
server_key: "3bd5b982f75672a2be7b2328ba2b0946"
server_key_iv: "4beba474262f8c7637f2421abb40b417"
server_key_version: "encryptionkey@0"
I20260131 03:13:39.206287 5067 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "3a25b27758f042edad8e61f3fa29147c"
format_stamp: "Formatted at 2026-01-31 03:13:39 on dist-test-slave-5nnp"
server_key: "3bd5b982f75672a2be7b2328ba2b0946"
server_key_iv: "4beba474262f8c7637f2421abb40b417"
server_key_version: "encryptionkey@0"
I20260131 03:13:39.207515 5067 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260131 03:13:39.208346 5081 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:39.208531 5067 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260131 03:13:39.208619 5067 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "3a25b27758f042edad8e61f3fa29147c"
format_stamp: "Formatted at 2026-01-31 03:13:39 on dist-test-slave-5nnp"
server_key: "3bd5b982f75672a2be7b2328ba2b0946"
server_key_iv: "4beba474262f8c7637f2421abb40b417"
server_key_version: "encryptionkey@0"
I20260131 03:13:39.208698 5067 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:39.224917 5067 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:39.225212 5067 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:39.225348 5067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:39.229419 5067 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:40155
I20260131 03:13:39.229465 5133 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:40155 every 8 connection(s)
I20260131 03:13:39.229768 5067 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:13:39.230352 5134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:39.232667 5134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a25b27758f042edad8e61f3fa29147c: Bootstrap starting.
I20260131 03:13:39.233197 5134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a25b27758f042edad8e61f3fa29147c: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:39.233404 5134 log.cc:826] T 00000000000000000000000000000000 P 3a25b27758f042edad8e61f3fa29147c: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:39.234016 5134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a25b27758f042edad8e61f3fa29147c: No bootstrap required, opened a new log
I20260131 03:13:39.235174 5134 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3a25b27758f042edad8e61f3fa29147c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a25b27758f042edad8e61f3fa29147c" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 40155 } }
I20260131 03:13:39.235289 5134 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3a25b27758f042edad8e61f3fa29147c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:39.235309 5134 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3a25b27758f042edad8e61f3fa29147c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a25b27758f042edad8e61f3fa29147c, State: Initialized, Role: FOLLOWER
I20260131 03:13:39.235456 5134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a25b27758f042edad8e61f3fa29147c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a25b27758f042edad8e61f3fa29147c" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 40155 } }
I20260131 03:13:39.235528 5134 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3a25b27758f042edad8e61f3fa29147c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:13:39.235551 5134 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3a25b27758f042edad8e61f3fa29147c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:13:39.235580 5134 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3a25b27758f042edad8e61f3fa29147c [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:39.236068 5134 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3a25b27758f042edad8e61f3fa29147c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a25b27758f042edad8e61f3fa29147c" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 40155 } }
I20260131 03:13:39.236161 5134 leader_election.cc:304] T 00000000000000000000000000000000 P 3a25b27758f042edad8e61f3fa29147c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a25b27758f042edad8e61f3fa29147c; no voters:
I20260131 03:13:39.236274 5134 leader_election.cc:290] T 00000000000000000000000000000000 P 3a25b27758f042edad8e61f3fa29147c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:13:39.236373 5137 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3a25b27758f042edad8e61f3fa29147c [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:13:39.236526 5137 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3a25b27758f042edad8e61f3fa29147c [term 1 LEADER]: Becoming Leader. State: Replica: 3a25b27758f042edad8e61f3fa29147c, State: Running, Role: LEADER
I20260131 03:13:39.236531 5134 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3a25b27758f042edad8e61f3fa29147c [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:13:39.236645 5137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a25b27758f042edad8e61f3fa29147c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a25b27758f042edad8e61f3fa29147c" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 40155 } }
I20260131 03:13:39.236995 5138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a25b27758f042edad8e61f3fa29147c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a25b27758f042edad8e61f3fa29147c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a25b27758f042edad8e61f3fa29147c" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 40155 } } }
I20260131 03:13:39.237092 5138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a25b27758f042edad8e61f3fa29147c [sys.catalog]: This master's current role is: LEADER
I20260131 03:13:39.237004 5139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a25b27758f042edad8e61f3fa29147c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a25b27758f042edad8e61f3fa29147c. Latest consensus state: current_term: 1 leader_uuid: "3a25b27758f042edad8e61f3fa29147c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a25b27758f042edad8e61f3fa29147c" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 40155 } } }
I20260131 03:13:39.237219 5139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a25b27758f042edad8e61f3fa29147c [sys.catalog]: This master's current role is: LEADER
I20260131 03:13:39.237298 5143 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:13:39.237995 5143 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:13:39.239687 5143 catalog_manager.cc:1357] Generated new cluster ID: 48e2f7d83aa84839a981b877a063a7e1
I20260131 03:13:39.239691 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 5067
I20260131 03:13:39.239741 5143 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:13:39.239792 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:13:39.239975 19792 external_mini_cluster.cc:1468] Setting key 11ff93a8dd7c5888945109029001236c
I20260131 03:13:39.244851 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.240458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60912 (local address 127.19.84.62:40155)
0131 03:13:39.240731 (+ 273us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:39.240737 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:39.240758 (+ 21us) server_negotiation.cc:408] Connection header received
0131 03:13:39.240853 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:39.240858 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:39.240941 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:39.241050 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:39.241538 (+ 488us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.242646 (+ 1108us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:39.243467 (+ 821us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.243689 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:39.244118 (+ 429us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:39.244122 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:39.244123 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:13:39.244365 (+ 242us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:39.244392 (+ 27us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:13:39.244399 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:39.244467 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:39.244570 (+ 103us) server_negotiation.cc:300] Negotiation successful
0131 03:13:39.244704 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":73,"threads_started":1}
I20260131 03:13:39.249858 5143 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:13:39.250618 5143 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:13:39.259011 5143 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 3a25b27758f042edad8e61f3fa29147c: Generated new TSK 0
I20260131 03:13:39.259193 5143 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:13:39.260882 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:40155
--builtin_ntp_servers=127.19.84.20:42215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260131 03:13:39.339650 5158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:39.339852 5158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:39.339900 5158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:39.341451 5158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:39.341524 5158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:39.341564 5158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:13:39.343080 5158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:42215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:40155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:39.343359 5158 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:39.343658 5158 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:39.346338 5164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:39.346387 5158 server_base.cc:1060] running on GCE node
W20260131 03:13:39.346323 5163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:39.346494 5166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:39.346688 5158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:39.346956 5158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:39.348106 5158 hybrid_clock.cc:648] HybridClock initialized: now 1769829219348094 us; error 28 us; skew 500 ppm
I20260131 03:13:39.349323 5158 webserver.cc:492] Webserver started at http://127.19.84.1:45745/ using document root <none> and password file <none>
I20260131 03:13:39.349552 5158 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:39.349592 5158 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:39.349705 5158 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:39.350679 5158 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "3817233009f3488f83d56b356e311109"
format_stamp: "Formatted at 2026-01-31 03:13:39 on dist-test-slave-5nnp"
server_key: "c8d6e8f2dd5c94633b9a13c3c3e590a9"
server_key_iv: "05b679a0f986df7434fb11236e8d8e00"
server_key_version: "encryptionkey@0"
I20260131 03:13:39.351008 5158 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "3817233009f3488f83d56b356e311109"
format_stamp: "Formatted at 2026-01-31 03:13:39 on dist-test-slave-5nnp"
server_key: "c8d6e8f2dd5c94633b9a13c3c3e590a9"
server_key_iv: "05b679a0f986df7434fb11236e8d8e00"
server_key_version: "encryptionkey@0"
I20260131 03:13:39.352309 5158 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260131 03:13:39.353060 5172 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:39.353221 5158 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260131 03:13:39.353291 5158 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "3817233009f3488f83d56b356e311109"
format_stamp: "Formatted at 2026-01-31 03:13:39 on dist-test-slave-5nnp"
server_key: "c8d6e8f2dd5c94633b9a13c3c3e590a9"
server_key_iv: "05b679a0f986df7434fb11236e8d8e00"
server_key_version: "encryptionkey@0"
I20260131 03:13:39.353372 5158 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:39.389022 5158 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:39.389305 5158 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:39.389442 5158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:39.389703 5158 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:13:39.390043 5158 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:13:39.390075 5158 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:39.390098 5158 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:13:39.390168 5158 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:39.397027 5158 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:41237
I20260131 03:13:39.397089 5285 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:41237 every 8 connection(s)
I20260131 03:13:39.397445 5158 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
I20260131 03:13:39.402007 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.398327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:60141 (local address 127.19.84.62:40155)
0131 03:13:39.398423 (+ 96us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:39.398426 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:39.398596 (+ 170us) server_negotiation.cc:408] Connection header received
0131 03:13:39.399132 (+ 536us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:39.399133 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:39.399147 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:39.399212 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:39.399710 (+ 498us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.400310 (+ 600us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:39.400912 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.401088 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:39.401664 (+ 576us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:39.401668 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:39.401668 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:13:39.401673 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:39.401695 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:13:39.401699 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:39.401763 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:39.401892 (+ 129us) server_negotiation.cc:300] Negotiation successful
0131 03:13:39.401930 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260131 03:13:39.402153 5288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.398238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:40155 (local address 127.19.84.1:60141)
0131 03:13:39.398486 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:39.398513 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:39.399032 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:39.399276 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:39.399280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:39.399289 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:13:39.399611 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:39.399613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.400428 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:39.400429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:39.400815 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:39.400817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.400961 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:39.401524 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:13:39.401524 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:13:39.401546 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:13:39.401563 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:13:39.401763 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:13:39.401770 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:13:39.401773 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:13:39.401895 (+ 122us) client_negotiation.cc:240] Negotiation successful
0131 03:13:39.401970 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":108,"threads_started":1}
I20260131 03:13:39.402711 5286 heartbeater.cc:344] Connected to a master server at 127.19.84.62:40155
I20260131 03:13:39.402791 5286 heartbeater.cc:461] Registering TS with master...
I20260131 03:13:39.403005 5286 heartbeater.cc:507] Master 127.19.84.62:40155 requested a full tablet report, sending...
I20260131 03:13:39.403518 5098 ts_manager.cc:194] Registered new tserver with Master: 3817233009f3488f83d56b356e311109 (127.19.84.1:41237)
I20260131 03:13:39.404184 5098 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.19.84.1:60141
I20260131 03:13:39.405941 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 5158
I20260131 03:13:39.406064 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:13:39.406253 19792 external_mini_cluster.cc:1468] Setting key e2fcc2d8f776be4911b039e9e9cfba83
I20260131 03:13:39.407171 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:40155
--builtin_ntp_servers=127.19.84.20:42215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260131 03:13:39.485562 5289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:39.485759 5289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:39.485800 5289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:39.487182 5289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:39.487258 5289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:39.487295 5289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:13:39.488790 5289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:42215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:40155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:39.489041 5289 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:39.489279 5289 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:39.491815 5295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:39.491811 5294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:39.491811 5297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:39.491998 5289 server_base.cc:1060] running on GCE node
I20260131 03:13:39.492221 5289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:39.492429 5289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:39.493561 5289 hybrid_clock.cc:648] HybridClock initialized: now 1769829219493539 us; error 37 us; skew 500 ppm
I20260131 03:13:39.494700 5289 webserver.cc:492] Webserver started at http://127.19.84.2:44503/ using document root <none> and password file <none>
I20260131 03:13:39.494949 5289 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:39.495008 5289 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:39.495096 5289 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:39.496043 5289 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "d033d9b7dd444594adaddd00867b291c"
format_stamp: "Formatted at 2026-01-31 03:13:39 on dist-test-slave-5nnp"
server_key: "d98b82c3f4d30418004345e3987b1efb"
server_key_iv: "11946da0994e346df97d0540fefcb073"
server_key_version: "encryptionkey@0"
I20260131 03:13:39.496313 5289 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "d033d9b7dd444594adaddd00867b291c"
format_stamp: "Formatted at 2026-01-31 03:13:39 on dist-test-slave-5nnp"
server_key: "d98b82c3f4d30418004345e3987b1efb"
server_key_iv: "11946da0994e346df97d0540fefcb073"
server_key_version: "encryptionkey@0"
I20260131 03:13:39.497538 5289 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260131 03:13:39.498327 5303 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:39.498598 5289 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:13:39.498675 5289 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "d033d9b7dd444594adaddd00867b291c"
format_stamp: "Formatted at 2026-01-31 03:13:39 on dist-test-slave-5nnp"
server_key: "d98b82c3f4d30418004345e3987b1efb"
server_key_iv: "11946da0994e346df97d0540fefcb073"
server_key_version: "encryptionkey@0"
I20260131 03:13:39.498735 5289 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:39.519327 5289 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:39.519609 5289 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:39.519698 5289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:39.519886 5289 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:13:39.520284 5289 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:13:39.520319 5289 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:39.520341 5289 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:13:39.520391 5289 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:39.526090 5289 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:39657
I20260131 03:13:39.526150 5416 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:39657 every 8 connection(s)
I20260131 03:13:39.526434 5289 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
I20260131 03:13:39.530714 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.527186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:54403 (local address 127.19.84.62:40155)
0131 03:13:39.527250 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:39.527270 (+ 20us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:39.527496 (+ 226us) server_negotiation.cc:408] Connection header received
0131 03:13:39.528042 (+ 546us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:39.528044 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:39.528056 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:39.528118 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:39.528571 (+ 453us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.529091 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:39.529673 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.529837 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:39.530406 (+ 569us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:39.530410 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:39.530411 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:13:39.530415 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:39.530431 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:13:39.530435 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:39.530505 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:39.530602 (+ 97us) server_negotiation.cc:300] Negotiation successful
0131 03:13:39.530639 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260131 03:13:39.530908 5419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.527164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:40155 (local address 127.19.84.2:54403)
0131 03:13:39.527359 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:39.527410 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:39.527941 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:39.528184 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:39.528187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:39.528195 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:13:39.528479 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:39.528481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.529191 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:39.529192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:39.529584 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:39.529586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.529736 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:39.530290 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:13:39.530290 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:13:39.530308 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:13:39.530325 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:13:39.530505 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:13:39.530510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:13:39.530513 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:13:39.530606 (+ 93us) client_negotiation.cc:240] Negotiation successful
0131 03:13:39.530680 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260131 03:13:39.531435 5417 heartbeater.cc:344] Connected to a master server at 127.19.84.62:40155
I20260131 03:13:39.531562 5417 heartbeater.cc:461] Registering TS with master...
I20260131 03:13:39.531769 5417 heartbeater.cc:507] Master 127.19.84.62:40155 requested a full tablet report, sending...
I20260131 03:13:39.532094 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 5289
I20260131 03:13:39.532222 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:13:39.532223 5098 ts_manager.cc:194] Registered new tserver with Master: d033d9b7dd444594adaddd00867b291c (127.19.84.2:39657)
I20260131 03:13:39.532428 19792 external_mini_cluster.cc:1468] Setting key f3a1a8e9def92e322a696fc9b25134d1
I20260131 03:13:39.532660 5098 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.19.84.2:54403
I20260131 03:13:39.533489 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:40155
--builtin_ntp_servers=127.19.84.20:42215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260131 03:13:39.612430 5420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:39.612630 5420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:39.612672 5420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:39.614089 5420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:39.614132 5420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:39.614157 5420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:13:39.615650 5420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:42215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:40155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:39.615825 5420 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:39.616020 5420 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:39.618662 5426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:39.618738 5425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:39.618778 5420 server_base.cc:1060] running on GCE node
W20260131 03:13:39.618664 5428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:39.619050 5420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:39.619236 5420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:39.620368 5420 hybrid_clock.cc:648] HybridClock initialized: now 1769829219620350 us; error 32 us; skew 500 ppm
I20260131 03:13:39.621487 5420 webserver.cc:492] Webserver started at http://127.19.84.3:45599/ using document root <none> and password file <none>
I20260131 03:13:39.621688 5420 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:39.621729 5420 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:39.621867 5420 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:39.622804 5420 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "4ce32595e9b34559b0a5e05ffd69fdf8"
format_stamp: "Formatted at 2026-01-31 03:13:39 on dist-test-slave-5nnp"
server_key: "421a929c4c16f69d0a63b4c8be21eaf2"
server_key_iv: "48ed0309d85205988a9e6438d66ed16c"
server_key_version: "encryptionkey@0"
I20260131 03:13:39.623131 5420 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "4ce32595e9b34559b0a5e05ffd69fdf8"
format_stamp: "Formatted at 2026-01-31 03:13:39 on dist-test-slave-5nnp"
server_key: "421a929c4c16f69d0a63b4c8be21eaf2"
server_key_iv: "48ed0309d85205988a9e6438d66ed16c"
server_key_version: "encryptionkey@0"
I20260131 03:13:39.624384 5420 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260131 03:13:39.625092 5434 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:39.625276 5420 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260131 03:13:39.625365 5420 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "4ce32595e9b34559b0a5e05ffd69fdf8"
format_stamp: "Formatted at 2026-01-31 03:13:39 on dist-test-slave-5nnp"
server_key: "421a929c4c16f69d0a63b4c8be21eaf2"
server_key_iv: "48ed0309d85205988a9e6438d66ed16c"
server_key_version: "encryptionkey@0"
I20260131 03:13:39.625444 5420 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:39.645969 5420 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:39.646260 5420 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:39.646394 5420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:39.646622 5420 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:13:39.646965 5420 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:13:39.647018 5420 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:39.647051 5420 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:13:39.647082 5420 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:39.653003 5420 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:46805
I20260131 03:13:39.653060 5547 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:46805 every 8 connection(s)
I20260131 03:13:39.653398 5420 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
I20260131 03:13:39.657521 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.654160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:53865 (local address 127.19.84.62:40155)
0131 03:13:39.654222 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:39.654224 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:39.654387 (+ 163us) server_negotiation.cc:408] Connection header received
0131 03:13:39.654882 (+ 495us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:39.654883 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:39.654898 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:39.654963 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:39.655468 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.655977 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:39.656497 (+ 520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.656603 (+ 106us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:39.657150 (+ 547us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:39.657154 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:39.657154 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:13:39.657158 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:39.657172 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:13:39.657175 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:39.657220 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:39.657430 (+ 210us) server_negotiation.cc:300] Negotiation successful
0131 03:13:39.657464 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260131 03:13:39.657636 5550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.654102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:40155 (local address 127.19.84.3:53865)
0131 03:13:39.654281 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:39.654311 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:39.654775 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:39.655026 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:39.655030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:39.655039 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:13:39.655332 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:39.655335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.656051 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:39.656052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:39.656426 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:39.656427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.656527 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:39.657040 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:13:39.657040 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:13:39.657059 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:13:39.657075 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:13:39.657339 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:13:39.657343 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:13:39.657346 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:13:39.657414 (+ 68us) client_negotiation.cc:240] Negotiation successful
0131 03:13:39.657474 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":85,"threads_started":1}
I20260131 03:13:39.657943 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 5420
I20260131 03:13:39.658046 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:13:39.658202 5548 heartbeater.cc:344] Connected to a master server at 127.19.84.62:40155
I20260131 03:13:39.658231 19792 external_mini_cluster.cc:1468] Setting key 6830b8b6663cdcb720499ee2940bc0d8
I20260131 03:13:39.658289 5548 heartbeater.cc:461] Registering TS with master...
I20260131 03:13:39.658540 5548 heartbeater.cc:507] Master 127.19.84.62:40155 requested a full tablet report, sending...
I20260131 03:13:39.658931 5098 ts_manager.cc:194] Registered new tserver with Master: 4ce32595e9b34559b0a5e05ffd69fdf8 (127.19.84.3:46805)
I20260131 03:13:39.659320 5098 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.19.84.3:53865
I20260131 03:13:39.670239 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:13:39.673923 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.671677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60924 (local address 127.19.84.62:40155)
0131 03:13:39.671741 (+ 64us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:39.671743 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:39.671803 (+ 60us) server_negotiation.cc:408] Connection header received
0131 03:13:39.671845 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:39.671846 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:39.671854 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:39.671901 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:39.672219 (+ 318us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.672662 (+ 443us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:39.673179 (+ 517us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.673283 (+ 104us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:39.673690 (+ 407us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:39.673693 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:39.673694 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:13:39.673697 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:39.673710 (+ 13us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:13:39.673713 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:39.673755 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:39.673846 (+ 91us) server_negotiation.cc:300] Negotiation successful
0131 03:13:39.673871 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260131 03:13:39.675421 5098 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:13:39.676683 5098 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:13:39.681532 5558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.678578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:46805 (local address 127.0.0.1:53758)
0131 03:13:39.678777 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:39.678808 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:39.678890 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:39.679189 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:39.679192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:39.679194 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:39.679462 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:39.679464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.680676 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:39.680677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:39.681248 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:39.681249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.681333 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:39.681371 (+ 38us) client_negotiation.cc:769] Sending connection context
0131 03:13:39.681424 (+ 53us) client_negotiation.cc:240] Negotiation successful
0131 03:13:39.681466 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":50,"threads_started":1}
I20260131 03:13:39.682107 5559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.678867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53758 (local address 127.19.84.3:46805)
0131 03:13:39.679021 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:39.679029 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:39.679039 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:13:39.679076 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:39.679079 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:39.679095 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:39.679168 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:39.679589 (+ 421us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.680588 (+ 999us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:39.681346 (+ 758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.681873 (+ 527us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:39.681966 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:39.681994 (+ 28us) server_negotiation.cc:300] Negotiation successful
0131 03:13:39.682027 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260131 03:13:39.682641 5556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.678618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:39657 (local address 127.0.0.1:43198)
0131 03:13:39.678862 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:39.678871 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:39.678913 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:39.679599 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:39.679600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:39.679601 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:39.679774 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:39.679775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.681554 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:39.681555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:39.682468 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:39.682471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.682516 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:39.682519 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:13:39.682541 (+ 22us) client_negotiation.cc:240] Negotiation successful
0131 03:13:39.682580 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":108,"threads_started":1}
I20260131 03:13:39.682777 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.678615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:41237 (local address 127.0.0.1:60570)
0131 03:13:39.678778 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:39.678809 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:39.678881 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:39.679405 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:39.679406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:39.679408 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:39.679644 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:39.679646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.681557 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:39.681558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:39.682523 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:39.682526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.682611 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:39.682616 (+ 5us) client_negotiation.cc:769] Sending connection context
0131 03:13:39.682649 (+ 33us) client_negotiation.cc:240] Negotiation successful
0131 03:13:39.682689 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260131 03:13:39.682721 5482 tablet_service.cc:1505] Processing CreateTablet for tablet dc1d2344881643b6b71c9b72615bc703 (DEFAULT_TABLE table=test-table [id=07456ce997aa4ba3a2ba135468e9169b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:13:39.683038 5482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1d2344881643b6b71c9b72615bc703. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:39.683539 5561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.679183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43198 (local address 127.19.84.2:39657)
0131 03:13:39.679347 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:39.679355 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:39.679364 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:13:39.679420 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:39.679422 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:39.679438 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:39.679848 (+ 410us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:39.679927 (+ 79us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.681442 (+ 1515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:39.682785 (+ 1343us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.683280 (+ 495us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:39.683356 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:39.683421 (+ 65us) server_negotiation.cc:300] Negotiation successful
0131 03:13:39.683462 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260131 03:13:39.683547 5560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.679015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.19.84.1:41237)
0131 03:13:39.679219 (+ 204us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:39.679227 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:39.679237 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:13:39.679275 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:39.679277 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:39.679294 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:39.679367 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:39.679911 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.681473 (+ 1562us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:39.682612 (+ 1139us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.683329 (+ 717us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:39.683435 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:39.683461 (+ 26us) server_negotiation.cc:300] Negotiation successful
0131 03:13:39.683490 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":91,"threads_started":1}
I20260131 03:13:39.684111 5220 tablet_service.cc:1505] Processing CreateTablet for tablet dc1d2344881643b6b71c9b72615bc703 (DEFAULT_TABLE table=test-table [id=07456ce997aa4ba3a2ba135468e9169b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:13:39.684394 5220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1d2344881643b6b71c9b72615bc703. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:39.685990 5562 tablet_bootstrap.cc:492] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8: Bootstrap starting.
I20260131 03:13:39.686554 5351 tablet_service.cc:1505] Processing CreateTablet for tablet dc1d2344881643b6b71c9b72615bc703 (DEFAULT_TABLE table=test-table [id=07456ce997aa4ba3a2ba135468e9169b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:13:39.686681 5562 tablet_bootstrap.cc:654] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:39.686864 5351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1d2344881643b6b71c9b72615bc703. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:39.686964 5562 log.cc:826] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:39.686968 5563 tablet_bootstrap.cc:492] T dc1d2344881643b6b71c9b72615bc703 P 3817233009f3488f83d56b356e311109: Bootstrap starting.
I20260131 03:13:39.687563 5563 tablet_bootstrap.cc:654] T dc1d2344881643b6b71c9b72615bc703 P 3817233009f3488f83d56b356e311109: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:39.687879 5562 tablet_bootstrap.cc:492] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8: No bootstrap required, opened a new log
I20260131 03:13:39.687955 5562 ts_tablet_manager.cc:1403] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:13:39.687969 5563 log.cc:826] T dc1d2344881643b6b71c9b72615bc703 P 3817233009f3488f83d56b356e311109: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:39.688611 5563 tablet_bootstrap.cc:492] T dc1d2344881643b6b71c9b72615bc703 P 3817233009f3488f83d56b356e311109: No bootstrap required, opened a new log
I20260131 03:13:39.688690 5563 ts_tablet_manager.cc:1403] T dc1d2344881643b6b71c9b72615bc703 P 3817233009f3488f83d56b356e311109: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:13:39.689661 5566 tablet_bootstrap.cc:492] T dc1d2344881643b6b71c9b72615bc703 P d033d9b7dd444594adaddd00867b291c: Bootstrap starting.
I20260131 03:13:39.689630 5562 raft_consensus.cc:359] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce32595e9b34559b0a5e05ffd69fdf8" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 46805 } } peers { permanent_uuid: "d033d9b7dd444594adaddd00867b291c" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39657 } } peers { permanent_uuid: "3817233009f3488f83d56b356e311109" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 41237 } }
I20260131 03:13:39.689816 5562 raft_consensus.cc:385] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:39.689877 5562 raft_consensus.cc:740] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ce32595e9b34559b0a5e05ffd69fdf8, State: Initialized, Role: FOLLOWER
I20260131 03:13:39.689987 5562 consensus_queue.cc:260] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce32595e9b34559b0a5e05ffd69fdf8" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 46805 } } peers { permanent_uuid: "d033d9b7dd444594adaddd00867b291c" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39657 } } peers { permanent_uuid: "3817233009f3488f83d56b356e311109" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 41237 } }
I20260131 03:13:39.690230 5566 tablet_bootstrap.cc:654] T dc1d2344881643b6b71c9b72615bc703 P d033d9b7dd444594adaddd00867b291c: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:39.690176 5563 raft_consensus.cc:359] T dc1d2344881643b6b71c9b72615bc703 P 3817233009f3488f83d56b356e311109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce32595e9b34559b0a5e05ffd69fdf8" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 46805 } } peers { permanent_uuid: "d033d9b7dd444594adaddd00867b291c" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39657 } } peers { permanent_uuid: "3817233009f3488f83d56b356e311109" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 41237 } }
I20260131 03:13:39.690256 5562 ts_tablet_manager.cc:1434] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:13:39.690271 5548 heartbeater.cc:499] Master 127.19.84.62:40155 was elected leader, sending a full tablet report...
I20260131 03:13:39.690408 5563 raft_consensus.cc:385] T dc1d2344881643b6b71c9b72615bc703 P 3817233009f3488f83d56b356e311109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:39.690580 5566 log.cc:826] T dc1d2344881643b6b71c9b72615bc703 P d033d9b7dd444594adaddd00867b291c: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:39.690505 5563 raft_consensus.cc:740] T dc1d2344881643b6b71c9b72615bc703 P 3817233009f3488f83d56b356e311109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3817233009f3488f83d56b356e311109, State: Initialized, Role: FOLLOWER
I20260131 03:13:39.690713 5563 consensus_queue.cc:260] T dc1d2344881643b6b71c9b72615bc703 P 3817233009f3488f83d56b356e311109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce32595e9b34559b0a5e05ffd69fdf8" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 46805 } } peers { permanent_uuid: "d033d9b7dd444594adaddd00867b291c" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39657 } } peers { permanent_uuid: "3817233009f3488f83d56b356e311109" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 41237 } }
I20260131 03:13:39.691056 5563 ts_tablet_manager.cc:1434] T dc1d2344881643b6b71c9b72615bc703 P 3817233009f3488f83d56b356e311109: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:13:39.691164 5286 heartbeater.cc:499] Master 127.19.84.62:40155 was elected leader, sending a full tablet report...
I20260131 03:13:39.691574 5566 tablet_bootstrap.cc:492] T dc1d2344881643b6b71c9b72615bc703 P d033d9b7dd444594adaddd00867b291c: No bootstrap required, opened a new log
I20260131 03:13:39.691646 5566 ts_tablet_manager.cc:1403] T dc1d2344881643b6b71c9b72615bc703 P d033d9b7dd444594adaddd00867b291c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:13:39.692690 5566 raft_consensus.cc:359] T dc1d2344881643b6b71c9b72615bc703 P d033d9b7dd444594adaddd00867b291c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce32595e9b34559b0a5e05ffd69fdf8" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 46805 } } peers { permanent_uuid: "d033d9b7dd444594adaddd00867b291c" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39657 } } peers { permanent_uuid: "3817233009f3488f83d56b356e311109" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 41237 } }
I20260131 03:13:39.692800 5566 raft_consensus.cc:385] T dc1d2344881643b6b71c9b72615bc703 P d033d9b7dd444594adaddd00867b291c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:39.692821 5566 raft_consensus.cc:740] T dc1d2344881643b6b71c9b72615bc703 P d033d9b7dd444594adaddd00867b291c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d033d9b7dd444594adaddd00867b291c, State: Initialized, Role: FOLLOWER
I20260131 03:13:39.692966 5566 consensus_queue.cc:260] T dc1d2344881643b6b71c9b72615bc703 P d033d9b7dd444594adaddd00867b291c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce32595e9b34559b0a5e05ffd69fdf8" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 46805 } } peers { permanent_uuid: "d033d9b7dd444594adaddd00867b291c" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39657 } } peers { permanent_uuid: "3817233009f3488f83d56b356e311109" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 41237 } }
I20260131 03:13:39.693293 5566 ts_tablet_manager.cc:1434] T dc1d2344881643b6b71c9b72615bc703 P d033d9b7dd444594adaddd00867b291c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260131 03:13:39.693360 5417 heartbeater.cc:499] Master 127.19.84.62:40155 was elected leader, sending a full tablet report...
I20260131 03:13:39.712692 5567 raft_consensus.cc:493] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:13:39.712813 5567 raft_consensus.cc:515] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce32595e9b34559b0a5e05ffd69fdf8" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 46805 } } peers { permanent_uuid: "d033d9b7dd444594adaddd00867b291c" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39657 } } peers { permanent_uuid: "3817233009f3488f83d56b356e311109" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 41237 } }
I20260131 03:13:39.713217 5567 leader_election.cc:290] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d033d9b7dd444594adaddd00867b291c (127.19.84.2:39657), 3817233009f3488f83d56b356e311109 (127.19.84.1:41237)
I20260131 03:13:39.715533 5550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.713398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:41237 (local address 127.19.84.3:36855)
0131 03:13:39.713455 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:39.713468 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:39.713513 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:39.713665 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:39.713666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:39.713668 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:39.713845 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:39.713846 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.714531 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:39.714532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:39.715268 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:39.715271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.715332 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:39.715337 (+ 5us) client_negotiation.cc:769] Sending connection context
0131 03:13:39.715436 (+ 99us) client_negotiation.cc:240] Negotiation successful
0131 03:13:39.715475 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260131 03:13:39.715952 5560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.713463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:36855 (local address 127.19.84.1:41237)
0131 03:13:39.713504 (+ 41us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:39.713506 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:39.713516 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:13:39.713573 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:39.713574 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:39.713586 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:39.713662 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:39.713937 (+ 275us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.714438 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:39.715345 (+ 907us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.715822 (+ 477us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:39.715830 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:39.715853 (+ 23us) server_negotiation.cc:300] Negotiation successful
0131 03:13:39.715882 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260131 03:13:39.716178 5571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.713402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:39657 (local address 127.19.84.3:60929)
0131 03:13:39.713624 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:39.713633 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:39.713719 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:39.713910 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:39.713911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:39.713912 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:39.714130 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:39.714132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.715035 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:39.715036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:39.715935 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:39.715938 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.716020 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:39.716024 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:13:39.716056 (+ 32us) client_negotiation.cc:240] Negotiation successful
0131 03:13:39.716093 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":15,"thread_start_us":128,"threads_started":1}
I20260131 03:13:39.716265 5240 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc1d2344881643b6b71c9b72615bc703" candidate_uuid: "4ce32595e9b34559b0a5e05ffd69fdf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3817233009f3488f83d56b356e311109" is_pre_election: true
I20260131 03:13:39.716434 5240 raft_consensus.cc:2468] T dc1d2344881643b6b71c9b72615bc703 P 3817233009f3488f83d56b356e311109 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ce32595e9b34559b0a5e05ffd69fdf8 in term 0.
I20260131 03:13:39.716687 5561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.713466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:60929 (local address 127.19.84.2:39657)
0131 03:13:39.713515 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:39.713517 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:39.713704 (+ 187us) server_negotiation.cc:408] Connection header received
0131 03:13:39.713802 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:39.713803 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:39.713843 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:39.713910 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:39.714220 (+ 310us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.714690 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:39.716054 (+ 1364us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.716580 (+ 526us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:39.716588 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:39.716616 (+ 28us) server_negotiation.cc:300] Negotiation successful
0131 03:13:39.716641 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260131 03:13:39.716684 5438 leader_election.cc:304] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3817233009f3488f83d56b356e311109, 4ce32595e9b34559b0a5e05ffd69fdf8; no voters:
I20260131 03:13:39.716818 5567 raft_consensus.cc:2804] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:13:39.716884 5567 raft_consensus.cc:493] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:13:39.716912 5567 raft_consensus.cc:3060] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:39.717072 5371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc1d2344881643b6b71c9b72615bc703" candidate_uuid: "4ce32595e9b34559b0a5e05ffd69fdf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d033d9b7dd444594adaddd00867b291c" is_pre_election: true
I20260131 03:13:39.717197 5371 raft_consensus.cc:2468] T dc1d2344881643b6b71c9b72615bc703 P d033d9b7dd444594adaddd00867b291c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ce32595e9b34559b0a5e05ffd69fdf8 in term 0.
I20260131 03:13:39.717638 5567 raft_consensus.cc:515] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce32595e9b34559b0a5e05ffd69fdf8" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 46805 } } peers { permanent_uuid: "d033d9b7dd444594adaddd00867b291c" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39657 } } peers { permanent_uuid: "3817233009f3488f83d56b356e311109" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 41237 } }
I20260131 03:13:39.717744 5567 leader_election.cc:290] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8 [CANDIDATE]: Term 1 election: Requested vote from peers d033d9b7dd444594adaddd00867b291c (127.19.84.2:39657), 3817233009f3488f83d56b356e311109 (127.19.84.1:41237)
I20260131 03:13:39.717917 5240 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc1d2344881643b6b71c9b72615bc703" candidate_uuid: "4ce32595e9b34559b0a5e05ffd69fdf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3817233009f3488f83d56b356e311109"
I20260131 03:13:39.717926 5371 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc1d2344881643b6b71c9b72615bc703" candidate_uuid: "4ce32595e9b34559b0a5e05ffd69fdf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d033d9b7dd444594adaddd00867b291c"
I20260131 03:13:39.717986 5240 raft_consensus.cc:3060] T dc1d2344881643b6b71c9b72615bc703 P 3817233009f3488f83d56b356e311109 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:39.717999 5371 raft_consensus.cc:3060] T dc1d2344881643b6b71c9b72615bc703 P d033d9b7dd444594adaddd00867b291c [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:39.718909 5371 raft_consensus.cc:2468] T dc1d2344881643b6b71c9b72615bc703 P d033d9b7dd444594adaddd00867b291c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ce32595e9b34559b0a5e05ffd69fdf8 in term 1.
I20260131 03:13:39.718981 5240 raft_consensus.cc:2468] T dc1d2344881643b6b71c9b72615bc703 P 3817233009f3488f83d56b356e311109 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ce32595e9b34559b0a5e05ffd69fdf8 in term 1.
I20260131 03:13:39.719148 5435 leader_election.cc:304] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ce32595e9b34559b0a5e05ffd69fdf8, d033d9b7dd444594adaddd00867b291c; no voters:
I20260131 03:13:39.719311 5567 raft_consensus.cc:2804] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:13:39.719538 5567 raft_consensus.cc:697] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8 [term 1 LEADER]: Becoming Leader. State: Replica: 4ce32595e9b34559b0a5e05ffd69fdf8, State: Running, Role: LEADER
I20260131 03:13:39.719650 5567 consensus_queue.cc:237] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce32595e9b34559b0a5e05ffd69fdf8" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 46805 } } peers { permanent_uuid: "d033d9b7dd444594adaddd00867b291c" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39657 } } peers { permanent_uuid: "3817233009f3488f83d56b356e311109" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 41237 } }
I20260131 03:13:39.720409 5098 catalog_manager.cc:5654] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ce32595e9b34559b0a5e05ffd69fdf8 (127.19.84.3). New cstate: current_term: 1 leader_uuid: "4ce32595e9b34559b0a5e05ffd69fdf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce32595e9b34559b0a5e05ffd69fdf8" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 46805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d033d9b7dd444594adaddd00867b291c" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3817233009f3488f83d56b356e311109" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 41237 } health_report { overall_health: UNKNOWN } } }
I20260131 03:13:39.728765 5559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.726652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53766 (local address 127.19.84.3:46805)
0131 03:13:39.726709 (+ 57us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:39.726711 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:39.726723 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:13:39.726764 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:39.726769 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:39.726785 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:39.726856 (+ 71us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:13:39.727100 (+ 244us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.727645 (+ 545us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:39.728251 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:39.728403 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:39.728434 (+ 31us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:13:39.728515 (+ 81us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:13:39.728605 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:39.728668 (+ 63us) server_negotiation.cc:300] Negotiation successful
0131 03:13:39.728693 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260131 03:13:39.730432 5240 raft_consensus.cc:1275] T dc1d2344881643b6b71c9b72615bc703 P 3817233009f3488f83d56b356e311109 [term 1 FOLLOWER]: Refusing update from remote peer 4ce32595e9b34559b0a5e05ffd69fdf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:13:39.730432 5371 raft_consensus.cc:1275] T dc1d2344881643b6b71c9b72615bc703 P d033d9b7dd444594adaddd00867b291c [term 1 FOLLOWER]: Refusing update from remote peer 4ce32595e9b34559b0a5e05ffd69fdf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:13:39.730805 5572 consensus_queue.cc:1048] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d033d9b7dd444594adaddd00867b291c" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:13:39.730975 5567 consensus_queue.cc:1048] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3817233009f3488f83d56b356e311109" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:13:39.733250 5575 mvcc.cc:204] Tried to move back new op lower bound from 7249220484013195264 to 7249220483972378624. Current Snapshot: MvccSnapshot[applied={T|T < 7249220484013195264}]
I20260131 03:13:39.734160 5577 mvcc.cc:204] Tried to move back new op lower bound from 7249220484013195264 to 7249220483972378624. Current Snapshot: MvccSnapshot[applied={T|T < 7249220484013195264}]
I20260131 03:13:39.735598 5097 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:13:39.735714 5097 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:13:39.736452 5097 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 3a25b27758f042edad8e61f3fa29147c: Sending DeleteTablet for 3 replicas of tablet dc1d2344881643b6b71c9b72615bc703
I20260131 03:13:39.736769 5351 tablet_service.cc:1552] Processing DeleteTablet for tablet dc1d2344881643b6b71c9b72615bc703 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:13:39 UTC) from {username='slave'} at 127.0.0.1:43198
I20260131 03:13:39.736763 5220 tablet_service.cc:1552] Processing DeleteTablet for tablet dc1d2344881643b6b71c9b72615bc703 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:13:39 UTC) from {username='slave'} at 127.0.0.1:60570
I20260131 03:13:39.737057 5482 tablet_service.cc:1552] Processing DeleteTablet for tablet dc1d2344881643b6b71c9b72615bc703 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:13:39 UTC) from {username='slave'} at 127.0.0.1:53758
I20260131 03:13:39.737236 5585 tablet_replica.cc:333] T dc1d2344881643b6b71c9b72615bc703 P 3817233009f3488f83d56b356e311109: stopping tablet replica
I20260131 03:13:39.737373 5584 tablet_replica.cc:333] T dc1d2344881643b6b71c9b72615bc703 P d033d9b7dd444594adaddd00867b291c: stopping tablet replica
I20260131 03:13:39.737382 5585 raft_consensus.cc:2243] T dc1d2344881643b6b71c9b72615bc703 P 3817233009f3488f83d56b356e311109 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:13:39.737396 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 5158
I20260131 03:13:39.737480 5585 raft_consensus.cc:2272] T dc1d2344881643b6b71c9b72615bc703 P 3817233009f3488f83d56b356e311109 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:13:39.740339 5584 raft_consensus.cc:2243] T dc1d2344881643b6b71c9b72615bc703 P d033d9b7dd444594adaddd00867b291c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:13:39.740485 5584 raft_consensus.cc:2272] T dc1d2344881643b6b71c9b72615bc703 P d033d9b7dd444594adaddd00867b291c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:13:39.741006 5584 ts_tablet_manager.cc:1916] T dc1d2344881643b6b71c9b72615bc703 P d033d9b7dd444594adaddd00867b291c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:13:39.742681 5584 ts_tablet_manager.cc:1929] T dc1d2344881643b6b71c9b72615bc703 P d033d9b7dd444594adaddd00867b291c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:13:39.742874 5584 log.cc:1199] T dc1d2344881643b6b71c9b72615bc703 P d033d9b7dd444594adaddd00867b291c: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-1/wal/wals/dc1d2344881643b6b71c9b72615bc703
I20260131 03:13:39.743134 5584 ts_tablet_manager.cc:1950] T dc1d2344881643b6b71c9b72615bc703 P d033d9b7dd444594adaddd00867b291c: Deleting consensus metadata
I20260131 03:13:39.743520 5586 tablet_replica.cc:333] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8: stopping tablet replica
I20260131 03:13:39.743604 5082 catalog_manager.cc:4985] TS d033d9b7dd444594adaddd00867b291c (127.19.84.2:39657): tablet dc1d2344881643b6b71c9b72615bc703 (table test-table [id=07456ce997aa4ba3a2ba135468e9169b]) successfully deleted
I20260131 03:13:39.743731 5586 raft_consensus.cc:2243] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:13:39.743880 5586 raft_consensus.cc:2272] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:13:39.744264 5586 ts_tablet_manager.cc:1916] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:13:39.745756 5586 ts_tablet_manager.cc:1929] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:13:39.745880 5586 log.cc:1199] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/dc1d2344881643b6b71c9b72615bc703
I20260131 03:13:39.746163 5586 ts_tablet_manager.cc:1950] T dc1d2344881643b6b71c9b72615bc703 P 4ce32595e9b34559b0a5e05ffd69fdf8: Deleting consensus metadata
W20260131 03:13:39.746461 5085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.84.1:41237 (error 108)
I20260131 03:13:39.746680 5084 catalog_manager.cc:4985] TS 4ce32595e9b34559b0a5e05ffd69fdf8 (127.19.84.3:46805): tablet dc1d2344881643b6b71c9b72615bc703 (table test-table [id=07456ce997aa4ba3a2ba135468e9169b]) successfully deleted
I20260131 03:13:39.746713 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 5289
I20260131 03:13:39.748289 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:39.746985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:41237 (local address 127.0.0.1:60580)
0131 03:13:39.748195 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:39.748250 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.1:41237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1184}
W20260131 03:13:39.748432 5085 catalog_manager.cc:4712] TS 3817233009f3488f83d56b356e311109 (127.19.84.1:41237): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc1d2344881643b6b71c9b72615bc703: Network error: Client connection negotiation failed: client connection to 127.19.84.1:41237: connect: Connection refused (error 111)
I20260131 03:13:39.753850 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 5420
I20260131 03:13:39.759012 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 5067
2026-01-31T03:13:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (670 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:13:39 dist-test-slave-5nnp krb5kdc[5590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:13:39 dist-test-slave-5nnp krb5kdc[5590](info): set up 2 sockets
Jan 31 03:13:39 dist-test-slave-5nnp krb5kdc[5590](info): commencing operation
krb5kdc: starting...
W20260131 03:13:41.759744 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.981s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:13:41 dist-test-slave-5nnp krb5kdc[5590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:13:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:13:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:13:41.897630 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:41399
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:41399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:13:41.977638 5606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:41.977897 5606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:41.977941 5606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:41.979496 5606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:13:41.979574 5606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:41.979599 5606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:41.979619 5606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:13:41.979650 5606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:13:41.981333 5606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:41399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:41399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:41.981621 5606 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:41.981837 5606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:41.985033 5614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:41.985040 5611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:41.985054 5612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:41.985222 5606 server_base.cc:1060] running on GCE node
I20260131 03:13:41.985601 5606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:41.986074 5606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:41.987251 5606 hybrid_clock.cc:648] HybridClock initialized: now 1769829221987233 us; error 53 us; skew 500 ppm
Jan 31 03:13:41 dist-test-slave-5nnp krb5kdc[5590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829221, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:13:41.990073 5606 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:13:41.990837 5606 webserver.cc:492] Webserver started at http://127.19.84.62:36835/ using document root <none> and password file <none>
I20260131 03:13:41.991075 5606 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:41.991134 5606 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:41.991288 5606 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:41.992354 5606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "0350d0dddf674a88830db6444b996592"
format_stamp: "Formatted at 2026-01-31 03:13:41 on dist-test-slave-5nnp"
server_key: "c54ccbc77d526cca5739f61775078489"
server_key_iv: "f8a6ddf080bbad1553878650b5b5e6e8"
server_key_version: "encryptionkey@0"
I20260131 03:13:41.992692 5606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "0350d0dddf674a88830db6444b996592"
format_stamp: "Formatted at 2026-01-31 03:13:41 on dist-test-slave-5nnp"
server_key: "c54ccbc77d526cca5739f61775078489"
server_key_iv: "f8a6ddf080bbad1553878650b5b5e6e8"
server_key_version: "encryptionkey@0"
I20260131 03:13:41.994016 5606 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260131 03:13:41.994951 5621 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:41.995190 5606 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:13:41.995287 5606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "0350d0dddf674a88830db6444b996592"
format_stamp: "Formatted at 2026-01-31 03:13:41 on dist-test-slave-5nnp"
server_key: "c54ccbc77d526cca5739f61775078489"
server_key_iv: "f8a6ddf080bbad1553878650b5b5e6e8"
server_key_version: "encryptionkey@0"
I20260131 03:13:41.995368 5606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:42.005986 5606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:42.007769 5606 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:42.007920 5606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:42.012153 5606 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:41399
I20260131 03:13:42.012200 5673 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:41399 every 8 connection(s)
I20260131 03:13:42.012496 5606 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:13:42.013067 5674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:42.015600 5674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0350d0dddf674a88830db6444b996592: Bootstrap starting.
I20260131 03:13:42.016165 5674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0350d0dddf674a88830db6444b996592: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:42.016415 5674 log.cc:826] T 00000000000000000000000000000000 P 0350d0dddf674a88830db6444b996592: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:42.017103 5674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0350d0dddf674a88830db6444b996592: No bootstrap required, opened a new log
I20260131 03:13:42.018263 5674 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0350d0dddf674a88830db6444b996592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0350d0dddf674a88830db6444b996592" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41399 } }
I20260131 03:13:42.018404 5674 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0350d0dddf674a88830db6444b996592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:42.018447 5674 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0350d0dddf674a88830db6444b996592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0350d0dddf674a88830db6444b996592, State: Initialized, Role: FOLLOWER
I20260131 03:13:42.018612 5674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0350d0dddf674a88830db6444b996592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0350d0dddf674a88830db6444b996592" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41399 } }
I20260131 03:13:42.018693 5674 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0350d0dddf674a88830db6444b996592 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:13:42.018738 5674 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0350d0dddf674a88830db6444b996592 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:13:42.018787 5674 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0350d0dddf674a88830db6444b996592 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:42.019306 5674 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0350d0dddf674a88830db6444b996592 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0350d0dddf674a88830db6444b996592" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41399 } }
I20260131 03:13:42.019455 5674 leader_election.cc:304] T 00000000000000000000000000000000 P 0350d0dddf674a88830db6444b996592 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0350d0dddf674a88830db6444b996592; no voters:
I20260131 03:13:42.019618 5674 leader_election.cc:290] T 00000000000000000000000000000000 P 0350d0dddf674a88830db6444b996592 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:13:42.019671 5677 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0350d0dddf674a88830db6444b996592 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:13:42.019862 5674 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0350d0dddf674a88830db6444b996592 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:13:42.019870 5677 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0350d0dddf674a88830db6444b996592 [term 1 LEADER]: Becoming Leader. State: Replica: 0350d0dddf674a88830db6444b996592, State: Running, Role: LEADER
I20260131 03:13:42.019984 5677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0350d0dddf674a88830db6444b996592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0350d0dddf674a88830db6444b996592" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41399 } }
I20260131 03:13:42.020356 5678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0350d0dddf674a88830db6444b996592 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0350d0dddf674a88830db6444b996592" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0350d0dddf674a88830db6444b996592" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41399 } } }
I20260131 03:13:42.020376 5679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0350d0dddf674a88830db6444b996592 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0350d0dddf674a88830db6444b996592. Latest consensus state: current_term: 1 leader_uuid: "0350d0dddf674a88830db6444b996592" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0350d0dddf674a88830db6444b996592" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 41399 } } }
I20260131 03:13:42.020521 5678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0350d0dddf674a88830db6444b996592 [sys.catalog]: This master's current role is: LEADER
I20260131 03:13:42.020537 5679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0350d0dddf674a88830db6444b996592 [sys.catalog]: This master's current role is: LEADER
I20260131 03:13:42.020757 5686 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:13:42.021222 5686 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:13:42.022239 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 5606
I20260131 03:13:42.022358 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:13:42.022547 19792 external_mini_cluster.cc:1468] Setting key ef66e1ed577846e07d13dc3d5f2daea3
I20260131 03:13:42.023026 5686 catalog_manager.cc:1357] Generated new cluster ID: 2289fdbb421b4003aff36f6b908129fc
I20260131 03:13:42.023075 5686 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Jan 31 03:13:42 dist-test-slave-5nnp krb5kdc[5590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:13:42.036419 5697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:42.023064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49220 (local address 127.19.84.62:41399)
0131 03:13:42.023295 (+ 231us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:42.023305 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:42.023326 (+ 21us) server_negotiation.cc:408] Connection header received
0131 03:13:42.023447 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:42.023453 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:42.023501 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:42.023620 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:42.024538 (+ 918us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.025771 (+ 1233us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:42.026808 (+ 1037us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.028266 (+ 1458us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:42.031621 (+ 3355us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:42.031628 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:42.031628 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:42.031654 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:42.034853 (+ 3199us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:42.035254 (+ 401us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:42.035257 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:42.035258 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:42.035332 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:42.035525 (+ 193us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:42.035526 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:42.035526 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:42.035806 (+ 280us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:42.035961 (+ 155us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:42.036085 (+ 124us) server_negotiation.cc:300] Negotiation successful
0131 03:13:42.036227 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":64,"threads_started":1}
I20260131 03:13:42.044874 5686 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:13:42.045455 5686 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:13:42.049492 5686 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 0350d0dddf674a88830db6444b996592: Generated new TSK 0
I20260131 03:13:42.049683 5686 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:13:42.105646 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:41399
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:13:42.189831 5702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:42.190018 5702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:42.190040 5702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:42.191560 5702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:42.191602 5702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:42.191629 5702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:13:42.193207 5702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:41399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:42.193414 5702 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:42.193635 5702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:42.196152 5708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:42.196204 5707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:42.196362 5710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:42.196450 5702 server_base.cc:1060] running on GCE node
I20260131 03:13:42.196615 5702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:42.196861 5702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:42.198012 5702 hybrid_clock.cc:648] HybridClock initialized: now 1769829222197997 us; error 32 us; skew 500 ppm
Jan 31 03:13:42 dist-test-slave-5nnp krb5kdc[5590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829222, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:13:42.200590 5702 init.cc:377] Logged in from keytab as kudu/127.19.84.1@KRBTEST.COM (short username kudu)
I20260131 03:13:42.201189 5702 webserver.cc:492] Webserver started at http://127.19.84.1:38007/ using document root <none> and password file <none>
I20260131 03:13:42.201372 5702 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:42.201412 5702 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:42.201509 5702 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:42.202617 5702 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "9473eefaa24e4b628981c56bb9c8b827"
format_stamp: "Formatted at 2026-01-31 03:13:42 on dist-test-slave-5nnp"
server_key: "592881c1aad8dba222a83d5b3bb6de26"
server_key_iv: "5ea5e4587c33ac46a5803311e9feae69"
server_key_version: "encryptionkey@0"
I20260131 03:13:42.202952 5702 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "9473eefaa24e4b628981c56bb9c8b827"
format_stamp: "Formatted at 2026-01-31 03:13:42 on dist-test-slave-5nnp"
server_key: "592881c1aad8dba222a83d5b3bb6de26"
server_key_iv: "5ea5e4587c33ac46a5803311e9feae69"
server_key_version: "encryptionkey@0"
I20260131 03:13:42.204445 5702 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260131 03:13:42.205195 5717 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:42.205379 5702 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:13:42.205464 5702 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "9473eefaa24e4b628981c56bb9c8b827"
format_stamp: "Formatted at 2026-01-31 03:13:42 on dist-test-slave-5nnp"
server_key: "592881c1aad8dba222a83d5b3bb6de26"
server_key_iv: "5ea5e4587c33ac46a5803311e9feae69"
server_key_version: "encryptionkey@0"
I20260131 03:13:42.205545 5702 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:42.243177 5702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:42.244989 5702 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:42.245147 5702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:42.245396 5702 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:13:42.245707 5702 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:13:42.245762 5702 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:42.245810 5702 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:13:42.245837 5702 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:42.251600 5702 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:44743
I20260131 03:13:42.251677 5830 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:44743 every 8 connection(s)
I20260131 03:13:42.251992 5702 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
Jan 31 03:13:42 dist-test-slave-5nnp krb5kdc[5590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829222, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:13:42.260391 5697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:42.252730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:56037 (local address 127.19.84.62:41399)
0131 03:13:42.252783 (+ 53us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:42.252785 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:42.252965 (+ 180us) server_negotiation.cc:408] Connection header received
0131 03:13:42.253465 (+ 500us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:42.253466 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:42.253482 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:42.253543 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:42.254305 (+ 762us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.254788 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:42.255356 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.255523 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:42.257851 (+ 2328us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:42.257856 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:42.257856 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:42.257863 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:42.259283 (+ 1420us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:42.259659 (+ 376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:42.259660 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:42.259660 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:42.259681 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:42.259858 (+ 177us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:42.259858 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:42.259858 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:42.259954 (+ 96us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:42.260003 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:42.260246 (+ 243us) server_negotiation.cc:300] Negotiation successful
0131 03:13:42.260323 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260131 03:13:42.260511 5833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:42.252681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41399 (local address 127.19.84.1:56037)
0131 03:13:42.252858 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:42.252893 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:42.253384 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:42.253608 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:42.253611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:42.253921 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:13:42.254225 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:42.254228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.254883 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:42.254885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:42.255281 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:42.255282 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.255416 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:42.256361 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:13:42.256378 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:13:42.257766 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:13:42.259364 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:42.259371 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:42.259394 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:42.259591 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:42.259741 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:42.259741 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:42.259741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:42.259808 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:42.260007 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:13:42.260011 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:13:42.260126 (+ 115us) client_negotiation.cc:769] Sending connection context
0131 03:13:42.260224 (+ 98us) client_negotiation.cc:240] Negotiation successful
0131 03:13:42.260328 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":95,"threads_started":1}
I20260131 03:13:42.261026 5831 heartbeater.cc:344] Connected to a master server at 127.19.84.62:41399
I20260131 03:13:42.261175 5831 heartbeater.cc:461] Registering TS with master...
I20260131 03:13:42.261380 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 5702
I20260131 03:13:42.261437 5831 heartbeater.cc:507] Master 127.19.84.62:41399 requested a full tablet report, sending...
I20260131 03:13:42.261497 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:13:42.261688 19792 external_mini_cluster.cc:1468] Setting key 7302abeb80f2f18808821771119cf40c
I20260131 03:13:42.261991 5638 ts_manager.cc:194] Registered new tserver with Master: 9473eefaa24e4b628981c56bb9c8b827 (127.19.84.1:44743)
I20260131 03:13:42.262884 5638 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.1@KRBTEST.COM'} at 127.19.84.1:56037
WARNING: no policy specified for kudu/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:13:42.308301 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:41399
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:13:42.388705 5838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:42.389005 5838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:42.389052 5838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:42.391009 5838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:42.391081 5838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:42.391125 5838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:13:42.392822 5838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:41399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:42.393095 5838 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:42.393292 5838 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:42.395807 5843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:42.395815 5844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:42.396169 5838 server_base.cc:1060] running on GCE node
W20260131 03:13:42.395828 5846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:42.396390 5838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:42.396616 5838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:42.397766 5838 hybrid_clock.cc:648] HybridClock initialized: now 1769829222397764 us; error 34 us; skew 500 ppm
Jan 31 03:13:42 dist-test-slave-5nnp krb5kdc[5590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829222, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:13:42.400079 5838 init.cc:377] Logged in from keytab as kudu/127.19.84.2@KRBTEST.COM (short username kudu)
I20260131 03:13:42.400599 5838 webserver.cc:492] Webserver started at http://127.19.84.2:36881/ using document root <none> and password file <none>
I20260131 03:13:42.400800 5838 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:42.400854 5838 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:42.400990 5838 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:42.401926 5838 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "99c8b01eb66549b199d76460356015f1"
format_stamp: "Formatted at 2026-01-31 03:13:42 on dist-test-slave-5nnp"
server_key: "7f959eb50cc590b48a74ae465280e9ef"
server_key_iv: "0afb2a503eb97c18d3803fb96f149179"
server_key_version: "encryptionkey@0"
I20260131 03:13:42.402303 5838 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "99c8b01eb66549b199d76460356015f1"
format_stamp: "Formatted at 2026-01-31 03:13:42 on dist-test-slave-5nnp"
server_key: "7f959eb50cc590b48a74ae465280e9ef"
server_key_iv: "0afb2a503eb97c18d3803fb96f149179"
server_key_version: "encryptionkey@0"
I20260131 03:13:42.403505 5838 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260131 03:13:42.404182 5853 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:42.404351 5838 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260131 03:13:42.404439 5838 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "99c8b01eb66549b199d76460356015f1"
format_stamp: "Formatted at 2026-01-31 03:13:42 on dist-test-slave-5nnp"
server_key: "7f959eb50cc590b48a74ae465280e9ef"
server_key_iv: "0afb2a503eb97c18d3803fb96f149179"
server_key_version: "encryptionkey@0"
I20260131 03:13:42.404515 5838 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:42.411568 5838 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:42.413528 5838 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:42.413671 5838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:42.413918 5838 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:13:42.414278 5838 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:13:42.414335 5838 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:42.414374 5838 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:13:42.414400 5838 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:42.419893 5838 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:42081
I20260131 03:13:42.419984 5966 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:42081 every 8 connection(s)
I20260131 03:13:42.420256 5838 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
I20260131 03:13:42.422546 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 5838
I20260131 03:13:42.422662 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:13:42.422839 19792 external_mini_cluster.cc:1468] Setting key 55bfb49f26efba9ea05e846c78aac3c5
Jan 31 03:13:42 dist-test-slave-5nnp krb5kdc[5590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829222, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:13:42.429435 5697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:42.421042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:54405 (local address 127.19.84.62:41399)
0131 03:13:42.421136 (+ 94us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:42.421139 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:42.421351 (+ 212us) server_negotiation.cc:408] Connection header received
0131 03:13:42.421846 (+ 495us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:42.421848 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:42.421860 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:42.421935 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:42.422669 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.423322 (+ 653us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:42.423973 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.424156 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:42.426447 (+ 2291us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:42.426451 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:42.426452 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:42.426459 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:42.427986 (+ 1527us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:42.428482 (+ 496us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:42.428482 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:42.428483 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:42.428504 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:42.428751 (+ 247us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:42.428751 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:42.428752 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:42.428838 (+ 86us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:42.428933 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:42.429281 (+ 348us) server_negotiation.cc:300] Negotiation successful
0131 03:13:42.429368 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260131 03:13:42.429620 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:42.421008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41399 (local address 127.19.84.2:54405)
0131 03:13:42.421245 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:42.421276 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:42.421752 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:42.422001 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:42.422004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:42.422302 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:13:42.422572 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:42.422575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.423460 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:42.423461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:42.423843 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:42.423845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.424016 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:42.424999 (+ 983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:13:42.425015 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:13:42.426351 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:13:42.428080 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:42.428087 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:42.428090 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:42.428402 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:42.428572 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:42.428573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:42.428574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:42.428683 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:42.428940 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:13:42.428946 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:13:42.429109 (+ 163us) client_negotiation.cc:769] Sending connection context
0131 03:13:42.429247 (+ 138us) client_negotiation.cc:240] Negotiation successful
0131 03:13:42.429417 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":88,"threads_started":1}
I20260131 03:13:42.430087 5967 heartbeater.cc:344] Connected to a master server at 127.19.84.62:41399
I20260131 03:13:42.430218 5967 heartbeater.cc:461] Registering TS with master...
I20260131 03:13:42.430437 5967 heartbeater.cc:507] Master 127.19.84.62:41399 requested a full tablet report, sending...
I20260131 03:13:42.430877 5638 ts_manager.cc:194] Registered new tserver with Master: 99c8b01eb66549b199d76460356015f1 (127.19.84.2:42081)
I20260131 03:13:42.431275 5638 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.2@KRBTEST.COM'} at 127.19.84.2:54405
WARNING: no policy specified for kudu/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:13:42.468354 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:41399
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:13:42.548267 5974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:42.548441 5974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:42.548465 5974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:42.550084 5974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:42.550133 5974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:42.550160 5974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:13:42.551860 5974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:41399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:42.552101 5974 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:42.552311 5974 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:42.554800 5982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:42.554777 5980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:42.554909 5974 server_base.cc:1060] running on GCE node
W20260131 03:13:42.554774 5979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:42.555167 5974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:42.555339 5974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:42.556505 5974 hybrid_clock.cc:648] HybridClock initialized: now 1769829222556493 us; error 27 us; skew 500 ppm
Jan 31 03:13:42 dist-test-slave-5nnp krb5kdc[5590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829222, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:13:42.558797 5974 init.cc:377] Logged in from keytab as kudu/127.19.84.3@KRBTEST.COM (short username kudu)
I20260131 03:13:42.559347 5974 webserver.cc:492] Webserver started at http://127.19.84.3:39979/ using document root <none> and password file <none>
I20260131 03:13:42.559574 5974 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:42.559631 5974 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:42.559747 5974 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:42.560662 5974 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "de9f4dc114014ea1a083bc6593f544d1"
format_stamp: "Formatted at 2026-01-31 03:13:42 on dist-test-slave-5nnp"
server_key: "97fc8191963c1c2bdb4dedaa9f4dec65"
server_key_iv: "6e8cbf32d4969b06c8f2f5abd49eebf3"
server_key_version: "encryptionkey@0"
I20260131 03:13:42.560984 5974 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "de9f4dc114014ea1a083bc6593f544d1"
format_stamp: "Formatted at 2026-01-31 03:13:42 on dist-test-slave-5nnp"
server_key: "97fc8191963c1c2bdb4dedaa9f4dec65"
server_key_iv: "6e8cbf32d4969b06c8f2f5abd49eebf3"
server_key_version: "encryptionkey@0"
I20260131 03:13:42.562140 5974 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260131 03:13:42.562777 5989 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:42.562964 5974 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260131 03:13:42.563048 5974 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "de9f4dc114014ea1a083bc6593f544d1"
format_stamp: "Formatted at 2026-01-31 03:13:42 on dist-test-slave-5nnp"
server_key: "97fc8191963c1c2bdb4dedaa9f4dec65"
server_key_iv: "6e8cbf32d4969b06c8f2f5abd49eebf3"
server_key_version: "encryptionkey@0"
I20260131 03:13:42.563125 5974 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:42.575300 5974 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:42.577293 5974 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:42.577438 5974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:42.577654 5974 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:13:42.577958 5974 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:13:42.578013 5974 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:42.578058 5974 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:13:42.578089 5974 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:42.583940 5974 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:35821
I20260131 03:13:42.584012 6102 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:35821 every 8 connection(s)
I20260131 03:13:42.584300 5974 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
Jan 31 03:13:42 dist-test-slave-5nnp krb5kdc[5590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829222, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:13:42.591933 5697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:42.584938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:49221 (local address 127.19.84.62:41399)
0131 03:13:42.585019 (+ 81us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:42.585021 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:42.585180 (+ 159us) server_negotiation.cc:408] Connection header received
0131 03:13:42.585610 (+ 430us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:42.585611 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:42.585622 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:42.585681 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:42.586343 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.586826 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:42.587409 (+ 583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.587522 (+ 113us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:42.589689 (+ 2167us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:42.589692 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:42.589693 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:42.589699 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:42.590808 (+ 1109us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:42.591147 (+ 339us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:42.591148 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:42.591148 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:42.591167 (+ 19us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:42.591368 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:42.591388 (+ 20us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:42.591388 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:42.591481 (+ 93us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:42.591545 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:42.591785 (+ 240us) server_negotiation.cc:300] Negotiation successful
0131 03:13:42.591869 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260131 03:13:42.591986 6105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:42.584905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:41399 (local address 127.19.84.3:49221)
0131 03:13:42.585077 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:42.585103 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:42.585536 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:42.585722 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:42.585725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:42.585992 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:13:42.586265 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:42.586267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.586960 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:42.586961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:42.587321 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:42.587322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.587476 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:42.588285 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:13:42.588299 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:13:42.589595 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:13:42.590876 (+ 1281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:42.590881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:42.590883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:42.591082 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:42.591223 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:42.591223 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:42.591224 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:42.591285 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:42.591546 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:13:42.591550 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:13:42.591651 (+ 101us) client_negotiation.cc:769] Sending connection context
0131 03:13:42.591747 (+ 96us) client_negotiation.cc:240] Negotiation successful
0131 03:13:42.591841 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
I20260131 03:13:42.592522 6103 heartbeater.cc:344] Connected to a master server at 127.19.84.62:41399
I20260131 03:13:42.592631 6103 heartbeater.cc:461] Registering TS with master...
I20260131 03:13:42.592604 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 5974
I20260131 03:13:42.592706 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:13:42.592829 6103 heartbeater.cc:507] Master 127.19.84.62:41399 requested a full tablet report, sending...
I20260131 03:13:42.592880 19792 external_mini_cluster.cc:1468] Setting key bdd6abbbbc163601f167c780b567c64f
I20260131 03:13:42.593237 5638 ts_manager.cc:194] Registered new tserver with Master: de9f4dc114014ea1a083bc6593f544d1 (127.19.84.3:35821)
I20260131 03:13:42.593726 5638 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:49221
I20260131 03:13:42.593770 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:13:42.600355 5697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:42.595084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49222 (local address 127.19.84.62:41399)
0131 03:13:42.595145 (+ 61us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:42.595146 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:42.595267 (+ 121us) server_negotiation.cc:408] Connection header received
0131 03:13:42.595316 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:42.595317 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:42.595324 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:42.595362 (+ 38us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:42.596037 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.596470 (+ 433us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:42.596994 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.597093 (+ 99us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:42.598061 (+ 968us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:42.598063 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:42.598064 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:42.598069 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:42.599314 (+ 1245us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:42.599709 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:42.599709 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:42.599710 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:42.599730 (+ 20us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:42.599925 (+ 195us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:42.599925 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:42.599926 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:42.600016 (+ 90us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:42.600102 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:42.600203 (+ 101us) server_negotiation.cc:300] Negotiation successful
0131 03:13:42.600247 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260131 03:13:42.601815 5638 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:13:42.603018 5638 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:13:42.608618 6116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:42.605159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:35821 (local address 127.0.0.1:42530)
0131 03:13:42.605472 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:42.605484 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:42.605708 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:42.606374 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:42.606375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:42.606376 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:42.606507 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:42.606508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.607364 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:42.607365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:42.608319 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:42.608321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.608397 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:42.608428 (+ 31us) client_negotiation.cc:769] Sending connection context
0131 03:13:42.608476 (+ 48us) client_negotiation.cc:240] Negotiation successful
0131 03:13:42.608515 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":4,"thread_start_us":62,"threads_started":1}
I20260131 03:13:42.608618 6114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:42.605159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:42081 (local address 127.0.0.1:33928)
0131 03:13:42.605350 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:42.605379 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:42.605461 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:42.605841 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:42.605843 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:42.605845 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:42.606109 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:42.606112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.607459 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:42.607460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:42.608338 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:42.608340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.608395 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:42.608428 (+ 33us) client_negotiation.cc:769] Sending connection context
0131 03:13:42.608476 (+ 48us) client_negotiation.cc:240] Negotiation successful
0131 03:13:42.608515 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":58,"threads_started":1}
I20260131 03:13:42.609254 6117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:42.605351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42530 (local address 127.19.84.3:35821)
0131 03:13:42.605684 (+ 333us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:42.605692 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:42.605702 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:13:42.606219 (+ 517us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:42.606223 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:42.606241 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:42.606304 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:42.606582 (+ 278us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.607278 (+ 696us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:42.608417 (+ 1139us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.609061 (+ 644us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:42.609108 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:42.609139 (+ 31us) server_negotiation.cc:300] Negotiation successful
0131 03:13:42.609177 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":79,"threads_started":1}
I20260131 03:13:42.609444 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:42.605496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33928 (local address 127.19.84.2:42081)
0131 03:13:42.605664 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:42.605672 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:42.605683 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:13:42.605723 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:42.605726 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:42.605742 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:42.605821 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:42.606179 (+ 358us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.607316 (+ 1137us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:42.608844 (+ 1528us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.609214 (+ 370us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:42.609245 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:42.609270 (+ 25us) server_negotiation.cc:300] Negotiation successful
0131 03:13:42.609296 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
I20260131 03:13:42.609681 6119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:42.605570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42432 (local address 127.19.84.1:44743)
0131 03:13:42.605726 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:42.605734 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:42.605876 (+ 142us) server_negotiation.cc:408] Connection header received
0131 03:13:42.605921 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:42.605929 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:42.605943 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:42.606022 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:42.606313 (+ 291us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.607450 (+ 1137us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:42.608689 (+ 1239us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.609503 (+ 814us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:42.609541 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:42.609569 (+ 28us) server_negotiation.cc:300] Negotiation successful
0131 03:13:42.609609 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260131 03:13:42.609822 6115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:42.605357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:44743 (local address 127.0.0.1:42432)
0131 03:13:42.605747 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:42.605757 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:42.605814 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:42.606023 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:42.606024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:42.606026 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:42.606227 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:42.606229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.607516 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:42.607517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:42.608456 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:42.608459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.608526 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:42.608530 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:13:42.608561 (+ 31us) client_negotiation.cc:240] Negotiation successful
0131 03:13:42.608599 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":75,"threads_started":1}
I20260131 03:13:42.609776 6037 tablet_service.cc:1505] Processing CreateTablet for tablet f7a59b8b7a4143c1803051d1257f82a8 (DEFAULT_TABLE table=test-table [id=c2d7feb6b4e04b7383b0a421a07cc83e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:13:42.610098 6037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7a59b8b7a4143c1803051d1257f82a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:42.610395 5765 tablet_service.cc:1505] Processing CreateTablet for tablet f7a59b8b7a4143c1803051d1257f82a8 (DEFAULT_TABLE table=test-table [id=c2d7feb6b4e04b7383b0a421a07cc83e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:13:42.610704 5765 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7a59b8b7a4143c1803051d1257f82a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:42.611920 5901 tablet_service.cc:1505] Processing CreateTablet for tablet f7a59b8b7a4143c1803051d1257f82a8 (DEFAULT_TABLE table=test-table [id=c2d7feb6b4e04b7383b0a421a07cc83e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:13:42.612178 5901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7a59b8b7a4143c1803051d1257f82a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:42.612808 6120 tablet_bootstrap.cc:492] T f7a59b8b7a4143c1803051d1257f82a8 P 9473eefaa24e4b628981c56bb9c8b827: Bootstrap starting.
I20260131 03:13:42.613101 6121 tablet_bootstrap.cc:492] T f7a59b8b7a4143c1803051d1257f82a8 P de9f4dc114014ea1a083bc6593f544d1: Bootstrap starting.
I20260131 03:13:42.613375 6120 tablet_bootstrap.cc:654] T f7a59b8b7a4143c1803051d1257f82a8 P 9473eefaa24e4b628981c56bb9c8b827: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:42.613662 6120 log.cc:826] T f7a59b8b7a4143c1803051d1257f82a8 P 9473eefaa24e4b628981c56bb9c8b827: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:42.613672 6121 tablet_bootstrap.cc:654] T f7a59b8b7a4143c1803051d1257f82a8 P de9f4dc114014ea1a083bc6593f544d1: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:42.613965 6121 log.cc:826] T f7a59b8b7a4143c1803051d1257f82a8 P de9f4dc114014ea1a083bc6593f544d1: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:42.614352 6120 tablet_bootstrap.cc:492] T f7a59b8b7a4143c1803051d1257f82a8 P 9473eefaa24e4b628981c56bb9c8b827: No bootstrap required, opened a new log
I20260131 03:13:42.614418 6120 ts_tablet_manager.cc:1403] T f7a59b8b7a4143c1803051d1257f82a8 P 9473eefaa24e4b628981c56bb9c8b827: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:13:42.614687 6121 tablet_bootstrap.cc:492] T f7a59b8b7a4143c1803051d1257f82a8 P de9f4dc114014ea1a083bc6593f544d1: No bootstrap required, opened a new log
I20260131 03:13:42.614748 6121 ts_tablet_manager.cc:1403] T f7a59b8b7a4143c1803051d1257f82a8 P de9f4dc114014ea1a083bc6593f544d1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:13:42.615139 6124 tablet_bootstrap.cc:492] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1: Bootstrap starting.
I20260131 03:13:42.615721 6124 tablet_bootstrap.cc:654] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:42.616040 6124 log.cc:826] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:42.616135 6121 raft_consensus.cc:359] T f7a59b8b7a4143c1803051d1257f82a8 P de9f4dc114014ea1a083bc6593f544d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c8b01eb66549b199d76460356015f1" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42081 } } peers { permanent_uuid: "de9f4dc114014ea1a083bc6593f544d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 35821 } } peers { permanent_uuid: "9473eefaa24e4b628981c56bb9c8b827" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 44743 } }
I20260131 03:13:42.616307 6121 raft_consensus.cc:385] T f7a59b8b7a4143c1803051d1257f82a8 P de9f4dc114014ea1a083bc6593f544d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:42.616343 6121 raft_consensus.cc:740] T f7a59b8b7a4143c1803051d1257f82a8 P de9f4dc114014ea1a083bc6593f544d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de9f4dc114014ea1a083bc6593f544d1, State: Initialized, Role: FOLLOWER
I20260131 03:13:42.616227 6120 raft_consensus.cc:359] T f7a59b8b7a4143c1803051d1257f82a8 P 9473eefaa24e4b628981c56bb9c8b827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c8b01eb66549b199d76460356015f1" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42081 } } peers { permanent_uuid: "de9f4dc114014ea1a083bc6593f544d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 35821 } } peers { permanent_uuid: "9473eefaa24e4b628981c56bb9c8b827" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 44743 } }
I20260131 03:13:42.616385 6120 raft_consensus.cc:385] T f7a59b8b7a4143c1803051d1257f82a8 P 9473eefaa24e4b628981c56bb9c8b827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:42.616415 6120 raft_consensus.cc:740] T f7a59b8b7a4143c1803051d1257f82a8 P 9473eefaa24e4b628981c56bb9c8b827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9473eefaa24e4b628981c56bb9c8b827, State: Initialized, Role: FOLLOWER
I20260131 03:13:42.616482 6121 consensus_queue.cc:260] T f7a59b8b7a4143c1803051d1257f82a8 P de9f4dc114014ea1a083bc6593f544d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c8b01eb66549b199d76460356015f1" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42081 } } peers { permanent_uuid: "de9f4dc114014ea1a083bc6593f544d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 35821 } } peers { permanent_uuid: "9473eefaa24e4b628981c56bb9c8b827" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 44743 } }
I20260131 03:13:42.616526 6120 consensus_queue.cc:260] T f7a59b8b7a4143c1803051d1257f82a8 P 9473eefaa24e4b628981c56bb9c8b827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c8b01eb66549b199d76460356015f1" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42081 } } peers { permanent_uuid: "de9f4dc114014ea1a083bc6593f544d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 35821 } } peers { permanent_uuid: "9473eefaa24e4b628981c56bb9c8b827" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 44743 } }
I20260131 03:13:42.616755 6103 heartbeater.cc:499] Master 127.19.84.62:41399 was elected leader, sending a full tablet report...
I20260131 03:13:42.616864 5831 heartbeater.cc:499] Master 127.19.84.62:41399 was elected leader, sending a full tablet report...
I20260131 03:13:42.617035 6120 ts_tablet_manager.cc:1434] T f7a59b8b7a4143c1803051d1257f82a8 P 9473eefaa24e4b628981c56bb9c8b827: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260131 03:13:42.617228 6124 tablet_bootstrap.cc:492] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1: No bootstrap required, opened a new log
I20260131 03:13:42.616748 6121 ts_tablet_manager.cc:1434] T f7a59b8b7a4143c1803051d1257f82a8 P de9f4dc114014ea1a083bc6593f544d1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:13:42.617307 6124 ts_tablet_manager.cc:1403] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260131 03:13:42.618835 6124 raft_consensus.cc:359] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c8b01eb66549b199d76460356015f1" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42081 } } peers { permanent_uuid: "de9f4dc114014ea1a083bc6593f544d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 35821 } } peers { permanent_uuid: "9473eefaa24e4b628981c56bb9c8b827" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 44743 } }
I20260131 03:13:42.618968 6124 raft_consensus.cc:385] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:42.619009 6124 raft_consensus.cc:740] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99c8b01eb66549b199d76460356015f1, State: Initialized, Role: FOLLOWER
I20260131 03:13:42.619102 6124 consensus_queue.cc:260] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c8b01eb66549b199d76460356015f1" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42081 } } peers { permanent_uuid: "de9f4dc114014ea1a083bc6593f544d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 35821 } } peers { permanent_uuid: "9473eefaa24e4b628981c56bb9c8b827" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 44743 } }
I20260131 03:13:42.619304 6124 ts_tablet_manager.cc:1434] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260131 03:13:42.619443 5967 heartbeater.cc:499] Master 127.19.84.62:41399 was elected leader, sending a full tablet report...
W20260131 03:13:42.670919 5968 tablet.cc:2378] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260131 03:13:42.752697 5832 tablet.cc:2378] T f7a59b8b7a4143c1803051d1257f82a8 P 9473eefaa24e4b628981c56bb9c8b827: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:13:42.762557 6128 raft_consensus.cc:493] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:13:42.762667 6128 raft_consensus.cc:515] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c8b01eb66549b199d76460356015f1" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42081 } } peers { permanent_uuid: "de9f4dc114014ea1a083bc6593f544d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 35821 } } peers { permanent_uuid: "9473eefaa24e4b628981c56bb9c8b827" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 44743 } }
I20260131 03:13:42.762962 6128 leader_election.cc:290] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de9f4dc114014ea1a083bc6593f544d1 (127.19.84.3:35821), 9473eefaa24e4b628981c56bb9c8b827 (127.19.84.1:44743)
I20260131 03:13:42.766167 5969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:42.763165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:35821 (local address 127.19.84.2:37773)
0131 03:13:42.763227 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:42.763242 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:42.763291 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:42.763656 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:42.763656 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:42.763658 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:42.763911 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:42.763913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.764875 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:42.764877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:42.765865 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:42.765866 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.765952 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:42.765957 (+ 5us) client_negotiation.cc:769] Sending connection context
0131 03:13:42.765994 (+ 37us) client_negotiation.cc:240] Negotiation successful
0131 03:13:42.766040 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260131 03:13:42.766178 6129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:42.763323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:44743 (local address 127.19.84.2:53131)
0131 03:13:42.763518 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:42.763527 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:42.763580 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:42.763712 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:42.763713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:42.763714 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:42.763970 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:42.763972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.764903 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:42.764904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:42.765864 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:42.765866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.765980 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:42.765983 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:13:42.766016 (+ 33us) client_negotiation.cc:240] Negotiation successful
0131 03:13:42.766045 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":144,"threads_started":1}
I20260131 03:13:42.766589 6117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:42.763414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:37773 (local address 127.19.84.3:35821)
0131 03:13:42.763483 (+ 69us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:42.763485 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:42.763497 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:13:42.763529 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:42.763530 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:42.763549 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:42.763639 (+ 90us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:42.763984 (+ 345us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.764771 (+ 787us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:42.765977 (+ 1206us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.766456 (+ 479us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:42.766464 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:42.766504 (+ 40us) server_negotiation.cc:300] Negotiation successful
0131 03:13:42.766533 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260131 03:13:42.766580 6119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:42.763415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:53131 (local address 127.19.84.1:44743)
0131 03:13:42.763489 (+ 74us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:42.763491 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:42.763604 (+ 113us) server_negotiation.cc:408] Connection header received
0131 03:13:42.763635 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:42.763636 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:42.763650 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:42.763702 (+ 52us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:42.764047 (+ 345us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.764771 (+ 724us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:42.765988 (+ 1217us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.766450 (+ 462us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:42.766458 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:42.766481 (+ 23us) server_negotiation.cc:300] Negotiation successful
0131 03:13:42.766517 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260131 03:13:42.766842 6057 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7a59b8b7a4143c1803051d1257f82a8" candidate_uuid: "99c8b01eb66549b199d76460356015f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de9f4dc114014ea1a083bc6593f544d1" is_pre_election: true
I20260131 03:13:42.766875 5785 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7a59b8b7a4143c1803051d1257f82a8" candidate_uuid: "99c8b01eb66549b199d76460356015f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9473eefaa24e4b628981c56bb9c8b827" is_pre_election: true
I20260131 03:13:42.767002 6057 raft_consensus.cc:2468] T f7a59b8b7a4143c1803051d1257f82a8 P de9f4dc114014ea1a083bc6593f544d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99c8b01eb66549b199d76460356015f1 in term 0.
I20260131 03:13:42.767036 5785 raft_consensus.cc:2468] T f7a59b8b7a4143c1803051d1257f82a8 P 9473eefaa24e4b628981c56bb9c8b827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99c8b01eb66549b199d76460356015f1 in term 0.
I20260131 03:13:42.767194 5857 leader_election.cc:304] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99c8b01eb66549b199d76460356015f1, de9f4dc114014ea1a083bc6593f544d1; no voters:
I20260131 03:13:42.767339 6128 raft_consensus.cc:2804] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:13:42.767449 6128 raft_consensus.cc:493] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:13:42.767490 6128 raft_consensus.cc:3060] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:42.768146 6128 raft_consensus.cc:515] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c8b01eb66549b199d76460356015f1" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42081 } } peers { permanent_uuid: "de9f4dc114014ea1a083bc6593f544d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 35821 } } peers { permanent_uuid: "9473eefaa24e4b628981c56bb9c8b827" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 44743 } }
I20260131 03:13:42.768317 6128 leader_election.cc:290] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1 [CANDIDATE]: Term 1 election: Requested vote from peers de9f4dc114014ea1a083bc6593f544d1 (127.19.84.3:35821), 9473eefaa24e4b628981c56bb9c8b827 (127.19.84.1:44743)
I20260131 03:13:42.768517 6057 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7a59b8b7a4143c1803051d1257f82a8" candidate_uuid: "99c8b01eb66549b199d76460356015f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de9f4dc114014ea1a083bc6593f544d1"
I20260131 03:13:42.768532 5785 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f7a59b8b7a4143c1803051d1257f82a8" candidate_uuid: "99c8b01eb66549b199d76460356015f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9473eefaa24e4b628981c56bb9c8b827"
I20260131 03:13:42.768612 6057 raft_consensus.cc:3060] T f7a59b8b7a4143c1803051d1257f82a8 P de9f4dc114014ea1a083bc6593f544d1 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:42.768611 5785 raft_consensus.cc:3060] T f7a59b8b7a4143c1803051d1257f82a8 P 9473eefaa24e4b628981c56bb9c8b827 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:42.769295 6057 raft_consensus.cc:2468] T f7a59b8b7a4143c1803051d1257f82a8 P de9f4dc114014ea1a083bc6593f544d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99c8b01eb66549b199d76460356015f1 in term 1.
I20260131 03:13:42.769368 5785 raft_consensus.cc:2468] T f7a59b8b7a4143c1803051d1257f82a8 P 9473eefaa24e4b628981c56bb9c8b827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99c8b01eb66549b199d76460356015f1 in term 1.
I20260131 03:13:42.769516 5857 leader_election.cc:304] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99c8b01eb66549b199d76460356015f1, de9f4dc114014ea1a083bc6593f544d1; no voters:
I20260131 03:13:42.769673 6128 raft_consensus.cc:2804] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:13:42.769891 6128 raft_consensus.cc:697] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1 [term 1 LEADER]: Becoming Leader. State: Replica: 99c8b01eb66549b199d76460356015f1, State: Running, Role: LEADER
I20260131 03:13:42.770009 6128 consensus_queue.cc:237] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c8b01eb66549b199d76460356015f1" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42081 } } peers { permanent_uuid: "de9f4dc114014ea1a083bc6593f544d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 35821 } } peers { permanent_uuid: "9473eefaa24e4b628981c56bb9c8b827" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 44743 } }
I20260131 03:13:42.770823 5638 catalog_manager.cc:5654] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99c8b01eb66549b199d76460356015f1 (127.19.84.2). New cstate: current_term: 1 leader_uuid: "99c8b01eb66549b199d76460356015f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c8b01eb66549b199d76460356015f1" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 42081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de9f4dc114014ea1a083bc6593f544d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 35821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9473eefaa24e4b628981c56bb9c8b827" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 44743 } health_report { overall_health: UNKNOWN } } }
I20260131 03:13:42.790490 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:42.788242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33934 (local address 127.19.84.2:42081)
0131 03:13:42.788304 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:42.788307 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:42.788316 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:13:42.788403 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:42.788404 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:42.788424 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:42.788491 (+ 67us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:13:42.788741 (+ 250us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.789215 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:42.789796 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:42.789960 (+ 164us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:42.790134 (+ 174us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:13:42.790229 (+ 95us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:13:42.790330 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:42.790403 (+ 73us) server_negotiation.cc:300] Negotiation successful
0131 03:13:42.790411 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260131 03:13:42.792215 6057 raft_consensus.cc:1275] T f7a59b8b7a4143c1803051d1257f82a8 P de9f4dc114014ea1a083bc6593f544d1 [term 1 FOLLOWER]: Refusing update from remote peer 99c8b01eb66549b199d76460356015f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:13:42.792259 5785 raft_consensus.cc:1275] T f7a59b8b7a4143c1803051d1257f82a8 P 9473eefaa24e4b628981c56bb9c8b827 [term 1 FOLLOWER]: Refusing update from remote peer 99c8b01eb66549b199d76460356015f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:13:42.792551 6128 consensus_queue.cc:1048] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de9f4dc114014ea1a083bc6593f544d1" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 35821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:13:42.792688 6130 consensus_queue.cc:1048] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9473eefaa24e4b628981c56bb9c8b827" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 44743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:13:42.795138 6134 mvcc.cc:204] Tried to move back new op lower bound from 7249220496554344448 to 7249220496466583552. Current Snapshot: MvccSnapshot[applied={T|T < 7249220496554344448}]
I20260131 03:13:42.797773 5638 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:13:42.797858 5638 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:13:42.798545 5638 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 0350d0dddf674a88830db6444b996592: Sending DeleteTablet for 3 replicas of tablet f7a59b8b7a4143c1803051d1257f82a8
I20260131 03:13:42.798873 6037 tablet_service.cc:1552] Processing DeleteTablet for tablet f7a59b8b7a4143c1803051d1257f82a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:13:42 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:42530
I20260131 03:13:42.798902 5901 tablet_service.cc:1552] Processing DeleteTablet for tablet f7a59b8b7a4143c1803051d1257f82a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:13:42 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:33928
I20260131 03:13:42.798913 5765 tablet_service.cc:1552] Processing DeleteTablet for tablet f7a59b8b7a4143c1803051d1257f82a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:13:42 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:42432
I20260131 03:13:42.799171 6145 tablet_replica.cc:333] T f7a59b8b7a4143c1803051d1257f82a8 P 9473eefaa24e4b628981c56bb9c8b827: stopping tablet replica
I20260131 03:13:42.799227 6143 tablet_replica.cc:333] T f7a59b8b7a4143c1803051d1257f82a8 P de9f4dc114014ea1a083bc6593f544d1: stopping tablet replica
I20260131 03:13:42.799257 6145 raft_consensus.cc:2243] T f7a59b8b7a4143c1803051d1257f82a8 P 9473eefaa24e4b628981c56bb9c8b827 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:13:42.799304 6143 raft_consensus.cc:2243] T f7a59b8b7a4143c1803051d1257f82a8 P de9f4dc114014ea1a083bc6593f544d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:13:42.799350 6145 raft_consensus.cc:2272] T f7a59b8b7a4143c1803051d1257f82a8 P 9473eefaa24e4b628981c56bb9c8b827 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:13:42.799585 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 5702
I20260131 03:13:42.799726 6145 ts_tablet_manager.cc:1916] T f7a59b8b7a4143c1803051d1257f82a8 P 9473eefaa24e4b628981c56bb9c8b827: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:13:42.799433 6143 raft_consensus.cc:2272] T f7a59b8b7a4143c1803051d1257f82a8 P de9f4dc114014ea1a083bc6593f544d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:13:42.800719 6143 ts_tablet_manager.cc:1916] T f7a59b8b7a4143c1803051d1257f82a8 P de9f4dc114014ea1a083bc6593f544d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:13:42.802703 6143 ts_tablet_manager.cc:1929] T f7a59b8b7a4143c1803051d1257f82a8 P de9f4dc114014ea1a083bc6593f544d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:13:42.802779 6143 log.cc:1199] T f7a59b8b7a4143c1803051d1257f82a8 P de9f4dc114014ea1a083bc6593f544d1: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/f7a59b8b7a4143c1803051d1257f82a8
I20260131 03:13:42.803066 6143 ts_tablet_manager.cc:1950] T f7a59b8b7a4143c1803051d1257f82a8 P de9f4dc114014ea1a083bc6593f544d1: Deleting consensus metadata
I20260131 03:13:42.800273 6144 tablet_replica.cc:333] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1: stopping tablet replica
I20260131 03:13:42.803463 6144 raft_consensus.cc:2243] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:13:42.803479 5625 catalog_manager.cc:4985] TS de9f4dc114014ea1a083bc6593f544d1 (127.19.84.3:35821): tablet f7a59b8b7a4143c1803051d1257f82a8 (table test-table [id=c2d7feb6b4e04b7383b0a421a07cc83e]) successfully deleted
I20260131 03:13:42.803581 6144 raft_consensus.cc:2272] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:13:42.803916 6144 ts_tablet_manager.cc:1916] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:13:42.805857 6144 ts_tablet_manager.cc:1929] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:13:42.805938 6144 log.cc:1199] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1769828849314684-19792-0/minicluster-data/ts-1/wal/wals/f7a59b8b7a4143c1803051d1257f82a8
W20260131 03:13:42.806149 5625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.84.1:44743 (error 108)
I20260131 03:13:42.806304 6144 ts_tablet_manager.cc:1950] T f7a59b8b7a4143c1803051d1257f82a8 P 99c8b01eb66549b199d76460356015f1: Deleting consensus metadata
I20260131 03:13:42.806358 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 5838
I20260131 03:13:42.807332 6115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:42.807201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:44743 (local address 127.0.0.1:42444)
0131 03:13:42.807265 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:42.807304 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.1:44743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260131 03:13:42.807514 5625 catalog_manager.cc:4712] TS 9473eefaa24e4b628981c56bb9c8b827 (127.19.84.1:44743): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f7a59b8b7a4143c1803051d1257f82a8: Network error: Client connection negotiation failed: client connection to 127.19.84.1:44743: connect: Connection refused (error 111)
I20260131 03:13:42.811583 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 5974
I20260131 03:13:42.811610 6115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:42.811502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:42081 (local address 127.0.0.1:33938)
0131 03:13:42.811553 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:42.811583 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.2:42081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260131 03:13:42.816563 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 5606
2026-01-31T03:13:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (3058 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-01-31T03:13:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:13:42Z Disabled control of system clock
I20260131 03:13:42.844015 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:46645
--webserver_interface=127.19.84.62
--webserver_port=0
--builtin_ntp_servers=127.19.84.20:42111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:46645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260131 03:13:42.922302 6150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:42.922508 6150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:42.922530 6150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:42.923998 6150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:13:42.924043 6150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:42.924057 6150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:42.924069 6150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:13:42.924081 6150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:13:42.925581 6150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:42111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:46645
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:46645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:42.925776 6150 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:42.925966 6150 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:42.928586 6158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:42.928611 6156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:42.928732 6150 server_base.cc:1060] running on GCE node
W20260131 03:13:42.928622 6155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:42.929221 6150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:42.929507 6150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:42.930691 6150 hybrid_clock.cc:648] HybridClock initialized: now 1769829222930671 us; error 45 us; skew 500 ppm
I20260131 03:13:42.931982 6150 webserver.cc:492] Webserver started at http://127.19.84.62:44431/ using document root <none> and password file <none>
I20260131 03:13:42.932219 6150 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:42.932282 6150 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:42.932427 6150 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:42.933410 6150 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "455e37b6c9bc4a798367852c604e3ce6"
format_stamp: "Formatted at 2026-01-31 03:13:42 on dist-test-slave-5nnp"
server_key: "9250e1441209dd522f0102a0cf294abe"
server_key_iv: "10d4ba771cce406e90930cc78bdda66c"
server_key_version: "encryptionkey@0"
I20260131 03:13:42.933740 6150 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "455e37b6c9bc4a798367852c604e3ce6"
format_stamp: "Formatted at 2026-01-31 03:13:42 on dist-test-slave-5nnp"
server_key: "9250e1441209dd522f0102a0cf294abe"
server_key_iv: "10d4ba771cce406e90930cc78bdda66c"
server_key_version: "encryptionkey@0"
I20260131 03:13:42.935104 6150 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260131 03:13:42.936175 6164 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:42.936439 6150 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:13:42.936535 6150 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "455e37b6c9bc4a798367852c604e3ce6"
format_stamp: "Formatted at 2026-01-31 03:13:42 on dist-test-slave-5nnp"
server_key: "9250e1441209dd522f0102a0cf294abe"
server_key_iv: "10d4ba771cce406e90930cc78bdda66c"
server_key_version: "encryptionkey@0"
I20260131 03:13:42.936621 6150 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:43.004122 6150 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:43.004451 6150 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:43.004590 6150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:43.008903 6150 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:46645
I20260131 03:13:43.008960 6216 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:46645 every 8 connection(s)
I20260131 03:13:43.009434 6150 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:13:43.010090 6217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:43.012586 6217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 455e37b6c9bc4a798367852c604e3ce6: Bootstrap starting.
I20260131 03:13:43.013186 6217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 455e37b6c9bc4a798367852c604e3ce6: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:43.013445 6217 log.cc:826] T 00000000000000000000000000000000 P 455e37b6c9bc4a798367852c604e3ce6: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:43.014191 6217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 455e37b6c9bc4a798367852c604e3ce6: No bootstrap required, opened a new log
I20260131 03:13:43.015529 6217 raft_consensus.cc:359] T 00000000000000000000000000000000 P 455e37b6c9bc4a798367852c604e3ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "455e37b6c9bc4a798367852c604e3ce6" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 46645 } }
I20260131 03:13:43.015661 6217 raft_consensus.cc:385] T 00000000000000000000000000000000 P 455e37b6c9bc4a798367852c604e3ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:43.015681 6217 raft_consensus.cc:740] T 00000000000000000000000000000000 P 455e37b6c9bc4a798367852c604e3ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 455e37b6c9bc4a798367852c604e3ce6, State: Initialized, Role: FOLLOWER
I20260131 03:13:43.015813 6217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 455e37b6c9bc4a798367852c604e3ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "455e37b6c9bc4a798367852c604e3ce6" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 46645 } }
I20260131 03:13:43.015908 6217 raft_consensus.cc:399] T 00000000000000000000000000000000 P 455e37b6c9bc4a798367852c604e3ce6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:13:43.015933 6217 raft_consensus.cc:493] T 00000000000000000000000000000000 P 455e37b6c9bc4a798367852c604e3ce6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:13:43.016000 6217 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 455e37b6c9bc4a798367852c604e3ce6 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:43.016736 6217 raft_consensus.cc:515] T 00000000000000000000000000000000 P 455e37b6c9bc4a798367852c604e3ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "455e37b6c9bc4a798367852c604e3ce6" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 46645 } }
I20260131 03:13:43.016945 6217 leader_election.cc:304] T 00000000000000000000000000000000 P 455e37b6c9bc4a798367852c604e3ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 455e37b6c9bc4a798367852c604e3ce6; no voters:
I20260131 03:13:43.017166 6217 leader_election.cc:290] T 00000000000000000000000000000000 P 455e37b6c9bc4a798367852c604e3ce6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:13:43.017261 6220 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 455e37b6c9bc4a798367852c604e3ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:13:43.017421 6220 raft_consensus.cc:697] T 00000000000000000000000000000000 P 455e37b6c9bc4a798367852c604e3ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 455e37b6c9bc4a798367852c604e3ce6, State: Running, Role: LEADER
I20260131 03:13:43.017499 6217 sys_catalog.cc:565] T 00000000000000000000000000000000 P 455e37b6c9bc4a798367852c604e3ce6 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:13:43.017534 6220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 455e37b6c9bc4a798367852c604e3ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "455e37b6c9bc4a798367852c604e3ce6" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 46645 } }
I20260131 03:13:43.017894 6221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 455e37b6c9bc4a798367852c604e3ce6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "455e37b6c9bc4a798367852c604e3ce6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "455e37b6c9bc4a798367852c604e3ce6" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 46645 } } }
I20260131 03:13:43.017894 6222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 455e37b6c9bc4a798367852c604e3ce6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 455e37b6c9bc4a798367852c604e3ce6. Latest consensus state: current_term: 1 leader_uuid: "455e37b6c9bc4a798367852c604e3ce6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "455e37b6c9bc4a798367852c604e3ce6" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 46645 } } }
I20260131 03:13:43.017993 6221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 455e37b6c9bc4a798367852c604e3ce6 [sys.catalog]: This master's current role is: LEADER
I20260131 03:13:43.017998 6222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 455e37b6c9bc4a798367852c604e3ce6 [sys.catalog]: This master's current role is: LEADER
I20260131 03:13:43.018203 6225 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:13:43.018779 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 6150
I20260131 03:13:43.018919 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:13:43.018942 6225 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:13:43.019218 19792 external_mini_cluster.cc:1468] Setting key b87acb6e3823f778052b288ae5036094
I20260131 03:13:43.020933 6225 catalog_manager.cc:1357] Generated new cluster ID: e33837b11d944f049435a51b149e4f56
I20260131 03:13:43.021010 6225 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:13:43.024636 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:43.019735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40256 (local address 127.19.84.62:46645)
0131 03:13:43.019986 (+ 251us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:43.019994 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:43.020053 (+ 59us) server_negotiation.cc:408] Connection header received
0131 03:13:43.020183 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:43.020189 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:43.020261 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:43.020423 (+ 162us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:43.021081 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.022392 (+ 1311us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:43.023207 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.023510 (+ 303us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:43.023926 (+ 416us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:43.023952 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:43.023953 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:13:43.024187 (+ 234us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:43.024220 (+ 33us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:13:43.024228 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:43.024296 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:43.024416 (+ 120us) server_negotiation.cc:300] Negotiation successful
0131 03:13:43.024475 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":126,"threads_started":1}
I20260131 03:13:43.030862 6225 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:13:43.031507 6225 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:13:43.036908 6225 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 455e37b6c9bc4a798367852c604e3ce6: Generated new TSK 0
I20260131 03:13:43.037086 6225 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:13:43.038985 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:46645
--builtin_ntp_servers=127.19.84.20:42111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260131 03:13:43.117596 6241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:43.117808 6241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:43.117847 6241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:43.119617 6241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:43.119688 6241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:43.119727 6241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:13:43.121279 6241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:42111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:46645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:43.121536 6241 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:43.121779 6241 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:43.124528 6247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:43.124531 6249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:43.124801 6246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:43.124961 6241 server_base.cc:1060] running on GCE node
I20260131 03:13:43.125129 6241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:43.125308 6241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:43.126458 6241 hybrid_clock.cc:648] HybridClock initialized: now 1769829223126434 us; error 40 us; skew 500 ppm
I20260131 03:13:43.127820 6241 webserver.cc:492] Webserver started at http://127.19.84.1:39707/ using document root <none> and password file <none>
I20260131 03:13:43.128047 6241 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:43.128087 6241 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:43.128201 6241 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:43.129201 6241 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "44cab2527b774847b56621d1879e370a"
format_stamp: "Formatted at 2026-01-31 03:13:43 on dist-test-slave-5nnp"
server_key: "1cac932dd96b05701a4f320a78d003f9"
server_key_iv: "9428549509093612c080cf8f124efc4a"
server_key_version: "encryptionkey@0"
I20260131 03:13:43.129710 6241 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "44cab2527b774847b56621d1879e370a"
format_stamp: "Formatted at 2026-01-31 03:13:43 on dist-test-slave-5nnp"
server_key: "1cac932dd96b05701a4f320a78d003f9"
server_key_iv: "9428549509093612c080cf8f124efc4a"
server_key_version: "encryptionkey@0"
I20260131 03:13:43.130998 6241 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260131 03:13:43.131865 6255 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:43.132078 6241 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260131 03:13:43.132177 6241 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "44cab2527b774847b56621d1879e370a"
format_stamp: "Formatted at 2026-01-31 03:13:43 on dist-test-slave-5nnp"
server_key: "1cac932dd96b05701a4f320a78d003f9"
server_key_iv: "9428549509093612c080cf8f124efc4a"
server_key_version: "encryptionkey@0"
I20260131 03:13:43.132266 6241 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:43.143270 6241 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:43.143618 6241 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:43.143759 6241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:43.144295 6241 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:13:43.144722 6241 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:13:43.144780 6241 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:43.144825 6241 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:13:43.144851 6241 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:43.152570 6241 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:39955
I20260131 03:13:43.152665 6368 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:39955 every 8 connection(s)
I20260131 03:13:43.152967 6241 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
I20260131 03:13:43.153781 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 6241
I20260131 03:13:43.153887 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:13:43.154104 19792 external_mini_cluster.cc:1468] Setting key 3686b907f3412f5a3065182052fa29d3
I20260131 03:13:43.155277 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:46645
--builtin_ntp_servers=127.19.84.20:42111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260131 03:13:43.158191 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:43.153911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:35103 (local address 127.19.84.62:46645)
0131 03:13:43.153995 (+ 84us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:43.153997 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:43.154009 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:13:43.154806 (+ 797us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:43.154808 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:43.154822 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:43.154910 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:43.155563 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.156356 (+ 793us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:43.157078 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.157269 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:43.157889 (+ 620us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:43.157894 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:43.157894 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:13:43.157900 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:43.157916 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:13:43.157920 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:43.157970 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:43.158081 (+ 111us) server_negotiation.cc:300] Negotiation successful
0131 03:13:43.158116 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260131 03:13:43.158334 6371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:43.153701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:46645 (local address 127.19.84.1:35103)
0131 03:13:43.153881 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:43.153912 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:43.154684 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:43.154971 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:43.154976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:43.154987 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:13:43.155455 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:43.155460 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.156470 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:43.156471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:43.156984 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:43.156987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.157150 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:43.157755 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:13:43.157756 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:13:43.157777 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:13:43.157793 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:13:43.157979 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:13:43.157983 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:13:43.157986 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:13:43.158072 (+ 86us) client_negotiation.cc:240] Negotiation successful
0131 03:13:43.158159 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260131 03:13:43.158799 6369 heartbeater.cc:344] Connected to a master server at 127.19.84.62:46645
I20260131 03:13:43.158900 6369 heartbeater.cc:461] Registering TS with master...
I20260131 03:13:43.159092 6369 heartbeater.cc:507] Master 127.19.84.62:46645 requested a full tablet report, sending...
I20260131 03:13:43.159581 6181 ts_manager.cc:194] Registered new tserver with Master: 44cab2527b774847b56621d1879e370a (127.19.84.1:39955)
I20260131 03:13:43.160241 6181 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.19.84.1:35103
W20260131 03:13:43.247009 6372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:43.247161 6372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:43.247182 6372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:43.248528 6372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:43.248565 6372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:43.248589 6372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:13:43.250111 6372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:42111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:46645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:43.250344 6372 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:43.250574 6372 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:43.253222 6377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:43.253259 6380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:43.253304 6372 server_base.cc:1060] running on GCE node
W20260131 03:13:43.253269 6378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:43.253569 6372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:43.253746 6372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:43.254884 6372 hybrid_clock.cc:648] HybridClock initialized: now 1769829223254851 us; error 51 us; skew 500 ppm
I20260131 03:13:43.256059 6372 webserver.cc:492] Webserver started at http://127.19.84.2:45449/ using document root <none> and password file <none>
I20260131 03:13:43.256291 6372 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:43.256356 6372 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:43.256495 6372 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:43.257426 6372 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "7fc9d8201f4c4142866c18fbbe09a108"
format_stamp: "Formatted at 2026-01-31 03:13:43 on dist-test-slave-5nnp"
server_key: "2ffc6776285834c3652fe2ae1d901e07"
server_key_iv: "bbfdea97e54a7481880227cbad98bd10"
server_key_version: "encryptionkey@0"
I20260131 03:13:43.257741 6372 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "7fc9d8201f4c4142866c18fbbe09a108"
format_stamp: "Formatted at 2026-01-31 03:13:43 on dist-test-slave-5nnp"
server_key: "2ffc6776285834c3652fe2ae1d901e07"
server_key_iv: "bbfdea97e54a7481880227cbad98bd10"
server_key_version: "encryptionkey@0"
I20260131 03:13:43.258939 6372 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260131 03:13:43.259704 6386 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:43.259879 6372 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:43.259966 6372 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "7fc9d8201f4c4142866c18fbbe09a108"
format_stamp: "Formatted at 2026-01-31 03:13:43 on dist-test-slave-5nnp"
server_key: "2ffc6776285834c3652fe2ae1d901e07"
server_key_iv: "bbfdea97e54a7481880227cbad98bd10"
server_key_version: "encryptionkey@0"
I20260131 03:13:43.260043 6372 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:43.270699 6372 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:43.270941 6372 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:43.271067 6372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:43.271260 6372 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:13:43.271737 6372 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:13:43.271804 6372 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:43.271839 6372 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:13:43.271869 6372 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:43.277763 6372 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:39083
I20260131 03:13:43.277818 6499 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:39083 every 8 connection(s)
I20260131 03:13:43.278131 6372 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
I20260131 03:13:43.280718 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 6372
I20260131 03:13:43.280856 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:13:43.281095 19792 external_mini_cluster.cc:1468] Setting key 05d64d5c02721ee94f05c88437ba342d
I20260131 03:13:43.282107 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:46645
--builtin_ntp_servers=127.19.84.20:42111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260131 03:13:43.287966 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:43.278990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:48423 (local address 127.19.84.62:46645)
0131 03:13:43.279080 (+ 90us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:43.279083 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:43.279248 (+ 165us) server_negotiation.cc:408] Connection header received
0131 03:13:43.279792 (+ 544us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:43.279794 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:43.279807 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:43.279861 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:43.280384 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.280950 (+ 566us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:43.281635 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.281803 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:43.287616 (+ 5813us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:43.287621 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:43.287622 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:13:43.287627 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:43.287652 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:13:43.287656 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:43.287724 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:43.287833 (+ 109us) server_negotiation.cc:300] Negotiation successful
0131 03:13:43.287885 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260131 03:13:43.288144 6502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:43.278851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:46645 (local address 127.19.84.2:48423)
0131 03:13:43.279138 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:43.279165 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:43.279679 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:43.279972 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:43.279976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:43.279985 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:13:43.280286 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:43.280289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.281047 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:43.281048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:43.281546 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:43.281548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.281695 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:43.287463 (+ 5768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:13:43.287464 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:13:43.287492 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:13:43.287514 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:13:43.287727 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:13:43.287734 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:13:43.287738 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:13:43.287837 (+ 99us) client_negotiation.cc:240] Negotiation successful
0131 03:13:43.287927 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":123,"threads_started":1}
I20260131 03:13:43.288676 6500 heartbeater.cc:344] Connected to a master server at 127.19.84.62:46645
I20260131 03:13:43.288792 6500 heartbeater.cc:461] Registering TS with master...
I20260131 03:13:43.289001 6500 heartbeater.cc:507] Master 127.19.84.62:46645 requested a full tablet report, sending...
I20260131 03:13:43.289386 6181 ts_manager.cc:194] Registered new tserver with Master: 7fc9d8201f4c4142866c18fbbe09a108 (127.19.84.2:39083)
I20260131 03:13:43.289772 6181 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.19.84.2:48423
W20260131 03:13:43.362895 6503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:43.363116 6503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:43.363148 6503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:43.364692 6503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:43.364765 6503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:43.364804 6503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:13:43.366426 6503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:42111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:46645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:43.366672 6503 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:43.366916 6503 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:43.369657 6509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:43.369714 6511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:43.369663 6508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:43.369869 6503 server_base.cc:1060] running on GCE node
I20260131 03:13:43.370038 6503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:43.370266 6503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:43.371448 6503 hybrid_clock.cc:648] HybridClock initialized: now 1769829223371424 us; error 42 us; skew 500 ppm
I20260131 03:13:43.372565 6503 webserver.cc:492] Webserver started at http://127.19.84.3:35335/ using document root <none> and password file <none>
I20260131 03:13:43.372771 6503 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:43.372812 6503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:43.372936 6503 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:43.373862 6503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "0d9d85d8ade240b9b5922ef108652037"
format_stamp: "Formatted at 2026-01-31 03:13:43 on dist-test-slave-5nnp"
server_key: "bbac63eff6cce9b68ad91fec3a08feb8"
server_key_iv: "168b815964ae6fbdc1e37cf3ce5f7424"
server_key_version: "encryptionkey@0"
I20260131 03:13:43.374215 6503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "0d9d85d8ade240b9b5922ef108652037"
format_stamp: "Formatted at 2026-01-31 03:13:43 on dist-test-slave-5nnp"
server_key: "bbac63eff6cce9b68ad91fec3a08feb8"
server_key_iv: "168b815964ae6fbdc1e37cf3ce5f7424"
server_key_version: "encryptionkey@0"
I20260131 03:13:43.375499 6503 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:13:43.376354 6517 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:43.376557 6503 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:13:43.376619 6503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "0d9d85d8ade240b9b5922ef108652037"
format_stamp: "Formatted at 2026-01-31 03:13:43 on dist-test-slave-5nnp"
server_key: "bbac63eff6cce9b68ad91fec3a08feb8"
server_key_iv: "168b815964ae6fbdc1e37cf3ce5f7424"
server_key_version: "encryptionkey@0"
I20260131 03:13:43.376703 6503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:43.384737 6503 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:43.384967 6503 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:43.385053 6503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:43.385277 6503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:13:43.385566 6503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:13:43.385596 6503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:43.385617 6503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:13:43.385663 6503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:43.391022 6503 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:42337
I20260131 03:13:43.391139 6630 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:42337 every 8 connection(s)
I20260131 03:13:43.391427 6503 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
I20260131 03:13:43.395332 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:43.392179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:52769 (local address 127.19.84.62:46645)
0131 03:13:43.392291 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:43.392293 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:43.392424 (+ 131us) server_negotiation.cc:408] Connection header received
0131 03:13:43.392915 (+ 491us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:43.392916 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:43.392931 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:43.392990 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:43.393439 (+ 449us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.393903 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:43.394455 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.394559 (+ 104us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:43.395089 (+ 530us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:43.395093 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:43.395093 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:13:43.395098 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:43.395111 (+ 13us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:13:43.395115 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:43.395158 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:43.395248 (+ 90us) server_negotiation.cc:300] Negotiation successful
0131 03:13:43.395277 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260131 03:13:43.395465 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:43.392111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:46645 (local address 127.19.84.3:52769)
0131 03:13:43.392324 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:43.392355 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:43.392801 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:43.393052 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:43.393056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:43.393064 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:13:43.393361 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:43.393363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.393981 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:43.393982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:43.394383 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:43.394384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.394493 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:43.394974 (+ 481us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:13:43.394975 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:13:43.394993 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:13:43.395009 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:13:43.395174 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:13:43.395177 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:13:43.395179 (+ 2us) client_negotiation.cc:769] Sending connection context
0131 03:13:43.395234 (+ 55us) client_negotiation.cc:240] Negotiation successful
0131 03:13:43.395289 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":71,"threads_started":1}
I20260131 03:13:43.395925 6631 heartbeater.cc:344] Connected to a master server at 127.19.84.62:46645
I20260131 03:13:43.396046 6631 heartbeater.cc:461] Registering TS with master...
I20260131 03:13:43.396245 6631 heartbeater.cc:507] Master 127.19.84.62:46645 requested a full tablet report, sending...
I20260131 03:13:43.396512 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 6503
I20260131 03:13:43.396644 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:13:43.396699 6181 ts_manager.cc:194] Registered new tserver with Master: 0d9d85d8ade240b9b5922ef108652037 (127.19.84.3:42337)
I20260131 03:13:43.396848 19792 external_mini_cluster.cc:1468] Setting key 918649c5dce6c39ca0f335c61022d492
I20260131 03:13:43.397207 6181 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.19.84.3:52769
I20260131 03:13:43.397778 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:13:43.401639 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:43.399148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.19.84.62:46645)
0131 03:13:43.399229 (+ 81us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:43.399232 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:43.399279 (+ 47us) server_negotiation.cc:408] Connection header received
0131 03:13:43.399314 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:43.399315 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:43.399322 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:43.399413 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:43.399720 (+ 307us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.400169 (+ 449us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:43.400686 (+ 517us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.400794 (+ 108us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:43.401305 (+ 511us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:43.401307 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:43.401308 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:13:43.401312 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:43.401325 (+ 13us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:13:43.401328 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:43.401408 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:43.401522 (+ 114us) server_negotiation.cc:300] Negotiation successful
0131 03:13:43.401536 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260131 03:13:43.403040 6181 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:13:43.404342 6181 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:13:43.409729 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:43.406449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:42337 (local address 127.0.0.1:53836)
0131 03:13:43.407053 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:43.407061 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:43.407151 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:43.407419 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:43.407420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:43.407422 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:43.407604 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:43.407606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.408711 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:43.408713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:43.409439 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:43.409441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.409528 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:43.409568 (+ 40us) client_negotiation.cc:769] Sending connection context
0131 03:13:43.409615 (+ 47us) client_negotiation.cc:240] Negotiation successful
0131 03:13:43.409657 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":62,"threads_started":1}
I20260131 03:13:43.410048 6639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:43.406242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:39955 (local address 127.0.0.1:39624)
0131 03:13:43.406360 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:43.406390 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:43.406469 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:43.406869 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:43.406871 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:43.406872 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:43.407242 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:43.407244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.408932 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:43.408932 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:43.409847 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:43.409851 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.409895 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:43.409898 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:13:43.409921 (+ 23us) client_negotiation.cc:240] Negotiation successful
0131 03:13:43.409957 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
I20260131 03:13:43.410270 6644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:43.406565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53836 (local address 127.19.84.3:42337)
0131 03:13:43.406764 (+ 199us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:43.406773 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:43.407212 (+ 439us) server_negotiation.cc:408] Connection header received
0131 03:13:43.407250 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:43.407252 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:43.407268 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:43.407328 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:43.407690 (+ 362us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.408622 (+ 932us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:43.409524 (+ 902us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.410021 (+ 497us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:43.410101 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:43.410156 (+ 55us) server_negotiation.cc:300] Negotiation successful
0131 03:13:43.410197 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":109,"threads_started":1}
I20260131 03:13:43.410779 6565 tablet_service.cc:1505] Processing CreateTablet for tablet 59c5c07caf4d44d492e10e084a194d95 (DEFAULT_TABLE table=test-table [id=d41c7bec54994f8c897ee5062c5aebb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:13:43.410903 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:43.406515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39624 (local address 127.19.84.1:39955)
0131 03:13:43.406697 (+ 182us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:43.406709 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:43.406720 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:13:43.406760 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:43.406762 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:43.406781 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:43.406863 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:43.407333 (+ 470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.408838 (+ 1505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:43.410208 (+ 1370us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.410690 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:43.410750 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:43.410780 (+ 30us) server_negotiation.cc:300] Negotiation successful
0131 03:13:43.410825 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":74,"threads_started":1}
I20260131 03:13:43.410919 6640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:43.406260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:39083 (local address 127.0.0.1:35302)
0131 03:13:43.406646 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:43.406654 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:43.406730 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:43.406957 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:43.406957 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:43.406959 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:43.407133 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:43.407135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.408925 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:43.408926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:43.409952 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:43.409953 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.410749 (+ 796us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:43.410753 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:13:43.410821 (+ 68us) client_negotiation.cc:240] Negotiation successful
0131 03:13:43.410858 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":83,"threads_started":1}
I20260131 03:13:43.411135 6565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59c5c07caf4d44d492e10e084a194d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:43.411339 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:43.406346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35302 (local address 127.19.84.2:39083)
0131 03:13:43.406769 (+ 423us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:43.406777 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:43.406786 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:13:43.406824 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:43.406826 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:43.406854 (+ 28us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:43.406966 (+ 112us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:43.407238 (+ 272us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.408822 (+ 1584us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:43.410040 (+ 1218us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.410641 (+ 601us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:43.410717 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:43.410859 (+ 142us) server_negotiation.cc:300] Negotiation successful
0131 03:13:43.410915 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":91,"threads_started":1}
I20260131 03:13:43.411453 6303 tablet_service.cc:1505] Processing CreateTablet for tablet 59c5c07caf4d44d492e10e084a194d95 (DEFAULT_TABLE table=test-table [id=d41c7bec54994f8c897ee5062c5aebb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:13:43.411747 6303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59c5c07caf4d44d492e10e084a194d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:43.411856 6434 tablet_service.cc:1505] Processing CreateTablet for tablet 59c5c07caf4d44d492e10e084a194d95 (DEFAULT_TABLE table=test-table [id=d41c7bec54994f8c897ee5062c5aebb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:13:43.412114 6434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59c5c07caf4d44d492e10e084a194d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:43.413587 6645 tablet_bootstrap.cc:492] T 59c5c07caf4d44d492e10e084a194d95 P 0d9d85d8ade240b9b5922ef108652037: Bootstrap starting.
I20260131 03:13:43.414256 6645 tablet_bootstrap.cc:654] T 59c5c07caf4d44d492e10e084a194d95 P 0d9d85d8ade240b9b5922ef108652037: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:43.414531 6646 tablet_bootstrap.cc:492] T 59c5c07caf4d44d492e10e084a194d95 P 44cab2527b774847b56621d1879e370a: Bootstrap starting.
I20260131 03:13:43.414572 6645 log.cc:826] T 59c5c07caf4d44d492e10e084a194d95 P 0d9d85d8ade240b9b5922ef108652037: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:43.414934 6647 tablet_bootstrap.cc:492] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108: Bootstrap starting.
I20260131 03:13:43.415256 6646 tablet_bootstrap.cc:654] T 59c5c07caf4d44d492e10e084a194d95 P 44cab2527b774847b56621d1879e370a: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:43.415409 6645 tablet_bootstrap.cc:492] T 59c5c07caf4d44d492e10e084a194d95 P 0d9d85d8ade240b9b5922ef108652037: No bootstrap required, opened a new log
I20260131 03:13:43.415479 6645 ts_tablet_manager.cc:1403] T 59c5c07caf4d44d492e10e084a194d95 P 0d9d85d8ade240b9b5922ef108652037: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:13:43.415581 6646 log.cc:826] T 59c5c07caf4d44d492e10e084a194d95 P 44cab2527b774847b56621d1879e370a: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:43.416286 6646 tablet_bootstrap.cc:492] T 59c5c07caf4d44d492e10e084a194d95 P 44cab2527b774847b56621d1879e370a: No bootstrap required, opened a new log
I20260131 03:13:43.416354 6646 ts_tablet_manager.cc:1403] T 59c5c07caf4d44d492e10e084a194d95 P 44cab2527b774847b56621d1879e370a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260131 03:13:43.417076 6645 raft_consensus.cc:359] T 59c5c07caf4d44d492e10e084a194d95 P 0d9d85d8ade240b9b5922ef108652037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cab2527b774847b56621d1879e370a" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39955 } } peers { permanent_uuid: "7fc9d8201f4c4142866c18fbbe09a108" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39083 } } peers { permanent_uuid: "0d9d85d8ade240b9b5922ef108652037" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 42337 } }
I20260131 03:13:43.417214 6645 raft_consensus.cc:385] T 59c5c07caf4d44d492e10e084a194d95 P 0d9d85d8ade240b9b5922ef108652037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:43.417253 6645 raft_consensus.cc:740] T 59c5c07caf4d44d492e10e084a194d95 P 0d9d85d8ade240b9b5922ef108652037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9d85d8ade240b9b5922ef108652037, State: Initialized, Role: FOLLOWER
I20260131 03:13:43.417348 6645 consensus_queue.cc:260] T 59c5c07caf4d44d492e10e084a194d95 P 0d9d85d8ade240b9b5922ef108652037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cab2527b774847b56621d1879e370a" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39955 } } peers { permanent_uuid: "7fc9d8201f4c4142866c18fbbe09a108" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39083 } } peers { permanent_uuid: "0d9d85d8ade240b9b5922ef108652037" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 42337 } }
I20260131 03:13:43.417578 6645 ts_tablet_manager.cc:1434] T 59c5c07caf4d44d492e10e084a194d95 P 0d9d85d8ade240b9b5922ef108652037: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:13:43.417712 6631 heartbeater.cc:499] Master 127.19.84.62:46645 was elected leader, sending a full tablet report...
I20260131 03:13:43.417960 6646 raft_consensus.cc:359] T 59c5c07caf4d44d492e10e084a194d95 P 44cab2527b774847b56621d1879e370a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cab2527b774847b56621d1879e370a" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39955 } } peers { permanent_uuid: "7fc9d8201f4c4142866c18fbbe09a108" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39083 } } peers { permanent_uuid: "0d9d85d8ade240b9b5922ef108652037" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 42337 } }
I20260131 03:13:43.418092 6646 raft_consensus.cc:385] T 59c5c07caf4d44d492e10e084a194d95 P 44cab2527b774847b56621d1879e370a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:43.418129 6646 raft_consensus.cc:740] T 59c5c07caf4d44d492e10e084a194d95 P 44cab2527b774847b56621d1879e370a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44cab2527b774847b56621d1879e370a, State: Initialized, Role: FOLLOWER
I20260131 03:13:43.418253 6647 tablet_bootstrap.cc:654] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:43.418239 6646 consensus_queue.cc:260] T 59c5c07caf4d44d492e10e084a194d95 P 44cab2527b774847b56621d1879e370a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cab2527b774847b56621d1879e370a" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39955 } } peers { permanent_uuid: "7fc9d8201f4c4142866c18fbbe09a108" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39083 } } peers { permanent_uuid: "0d9d85d8ade240b9b5922ef108652037" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 42337 } }
I20260131 03:13:43.418484 6646 ts_tablet_manager.cc:1434] T 59c5c07caf4d44d492e10e084a194d95 P 44cab2527b774847b56621d1879e370a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260131 03:13:43.418594 6369 heartbeater.cc:499] Master 127.19.84.62:46645 was elected leader, sending a full tablet report...
I20260131 03:13:43.418540 6647 log.cc:826] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:43.419519 6647 tablet_bootstrap.cc:492] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108: No bootstrap required, opened a new log
I20260131 03:13:43.419591 6647 ts_tablet_manager.cc:1403] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260131 03:13:43.420775 6647 raft_consensus.cc:359] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cab2527b774847b56621d1879e370a" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39955 } } peers { permanent_uuid: "7fc9d8201f4c4142866c18fbbe09a108" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39083 } } peers { permanent_uuid: "0d9d85d8ade240b9b5922ef108652037" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 42337 } }
I20260131 03:13:43.420876 6647 raft_consensus.cc:385] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:43.420923 6647 raft_consensus.cc:740] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fc9d8201f4c4142866c18fbbe09a108, State: Initialized, Role: FOLLOWER
I20260131 03:13:43.421015 6647 consensus_queue.cc:260] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cab2527b774847b56621d1879e370a" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39955 } } peers { permanent_uuid: "7fc9d8201f4c4142866c18fbbe09a108" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39083 } } peers { permanent_uuid: "0d9d85d8ade240b9b5922ef108652037" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 42337 } }
I20260131 03:13:43.421234 6647 ts_tablet_manager.cc:1434] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:13:43.421310 6500 heartbeater.cc:499] Master 127.19.84.62:46645 was elected leader, sending a full tablet report...
W20260131 03:13:43.528672 6501 tablet.cc:2378] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260131 03:13:43.641932 6632 tablet.cc:2378] T 59c5c07caf4d44d492e10e084a194d95 P 0d9d85d8ade240b9b5922ef108652037: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260131 03:13:43.653717 6370 tablet.cc:2378] T 59c5c07caf4d44d492e10e084a194d95 P 44cab2527b774847b56621d1879e370a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:13:43.702924 6653 raft_consensus.cc:493] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:13:43.703074 6653 raft_consensus.cc:515] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cab2527b774847b56621d1879e370a" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39955 } } peers { permanent_uuid: "7fc9d8201f4c4142866c18fbbe09a108" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39083 } } peers { permanent_uuid: "0d9d85d8ade240b9b5922ef108652037" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 42337 } }
I20260131 03:13:43.703478 6653 leader_election.cc:290] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44cab2527b774847b56621d1879e370a (127.19.84.1:39955), 0d9d85d8ade240b9b5922ef108652037 (127.19.84.3:42337)
I20260131 03:13:43.705955 6502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:43.703683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:39955 (local address 127.19.84.2:55459)
0131 03:13:43.703728 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:43.703742 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:43.703789 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:43.703995 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:43.703996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:43.703999 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:43.704182 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:43.704184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.705065 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:43.705066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:43.705791 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:43.705793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.705847 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:43.705850 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:13:43.705873 (+ 23us) client_negotiation.cc:240] Negotiation successful
0131 03:13:43.705898 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260131 03:13:43.706465 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:43.703765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:55459 (local address 127.19.84.1:39955)
0131 03:13:43.703838 (+ 73us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:43.703841 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:43.703851 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:13:43.703882 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:43.703883 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:43.703898 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:43.703962 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:43.704294 (+ 332us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.704994 (+ 700us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:43.705876 (+ 882us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.706316 (+ 440us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:43.706324 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:43.706346 (+ 22us) server_negotiation.cc:300] Negotiation successful
0131 03:13:43.706375 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260131 03:13:43.706624 6654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:43.703683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:42337 (local address 127.19.84.2:33251)
0131 03:13:43.704093 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:43.704105 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:43.704164 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:43.704322 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:43.704323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:43.704325 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:43.704728 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:43.704730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.705563 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:43.705564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:43.706391 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:43.706394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.706468 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:43.706472 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:13:43.706505 (+ 33us) client_negotiation.cc:240] Negotiation successful
0131 03:13:43.706546 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":159,"thread_start_us":192,"threads_started":1}
I20260131 03:13:43.706835 6323 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59c5c07caf4d44d492e10e084a194d95" candidate_uuid: "7fc9d8201f4c4142866c18fbbe09a108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44cab2527b774847b56621d1879e370a" is_pre_election: true
I20260131 03:13:43.707031 6323 raft_consensus.cc:2468] T 59c5c07caf4d44d492e10e084a194d95 P 44cab2527b774847b56621d1879e370a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fc9d8201f4c4142866c18fbbe09a108 in term 0.
I20260131 03:13:43.707072 6644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:43.703763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:33251 (local address 127.19.84.3:42337)
0131 03:13:43.703828 (+ 65us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:43.703831 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:43.704184 (+ 353us) server_negotiation.cc:408] Connection header received
0131 03:13:43.704219 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:43.704220 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:43.704234 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:43.704298 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:43.704807 (+ 509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.705455 (+ 648us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:43.706478 (+ 1023us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.706946 (+ 468us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:43.706954 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:43.706978 (+ 24us) server_negotiation.cc:300] Negotiation successful
0131 03:13:43.707006 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260131 03:13:43.707281 6390 leader_election.cc:304] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44cab2527b774847b56621d1879e370a, 7fc9d8201f4c4142866c18fbbe09a108; no voters:
I20260131 03:13:43.707441 6653 raft_consensus.cc:2804] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:13:43.707535 6653 raft_consensus.cc:493] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:13:43.707572 6653 raft_consensus.cc:3060] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:43.707525 6585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59c5c07caf4d44d492e10e084a194d95" candidate_uuid: "7fc9d8201f4c4142866c18fbbe09a108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9d85d8ade240b9b5922ef108652037" is_pre_election: true
I20260131 03:13:43.707648 6585 raft_consensus.cc:2468] T 59c5c07caf4d44d492e10e084a194d95 P 0d9d85d8ade240b9b5922ef108652037 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fc9d8201f4c4142866c18fbbe09a108 in term 0.
I20260131 03:13:43.708350 6653 raft_consensus.cc:515] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cab2527b774847b56621d1879e370a" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39955 } } peers { permanent_uuid: "7fc9d8201f4c4142866c18fbbe09a108" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39083 } } peers { permanent_uuid: "0d9d85d8ade240b9b5922ef108652037" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 42337 } }
I20260131 03:13:43.708494 6653 leader_election.cc:290] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108 [CANDIDATE]: Term 1 election: Requested vote from peers 44cab2527b774847b56621d1879e370a (127.19.84.1:39955), 0d9d85d8ade240b9b5922ef108652037 (127.19.84.3:42337)
I20260131 03:13:43.708678 6585 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59c5c07caf4d44d492e10e084a194d95" candidate_uuid: "7fc9d8201f4c4142866c18fbbe09a108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9d85d8ade240b9b5922ef108652037"
I20260131 03:13:43.708678 6323 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59c5c07caf4d44d492e10e084a194d95" candidate_uuid: "7fc9d8201f4c4142866c18fbbe09a108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44cab2527b774847b56621d1879e370a"
I20260131 03:13:43.708761 6585 raft_consensus.cc:3060] T 59c5c07caf4d44d492e10e084a194d95 P 0d9d85d8ade240b9b5922ef108652037 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:43.708761 6323 raft_consensus.cc:3060] T 59c5c07caf4d44d492e10e084a194d95 P 44cab2527b774847b56621d1879e370a [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:43.709609 6323 raft_consensus.cc:2468] T 59c5c07caf4d44d492e10e084a194d95 P 44cab2527b774847b56621d1879e370a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fc9d8201f4c4142866c18fbbe09a108 in term 1.
I20260131 03:13:43.709609 6585 raft_consensus.cc:2468] T 59c5c07caf4d44d492e10e084a194d95 P 0d9d85d8ade240b9b5922ef108652037 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fc9d8201f4c4142866c18fbbe09a108 in term 1.
I20260131 03:13:43.709817 6388 leader_election.cc:304] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9d85d8ade240b9b5922ef108652037, 7fc9d8201f4c4142866c18fbbe09a108; no voters:
I20260131 03:13:43.709960 6653 raft_consensus.cc:2804] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:13:43.710086 6653 raft_consensus.cc:697] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108 [term 1 LEADER]: Becoming Leader. State: Replica: 7fc9d8201f4c4142866c18fbbe09a108, State: Running, Role: LEADER
I20260131 03:13:43.710161 6653 consensus_queue.cc:237] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cab2527b774847b56621d1879e370a" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39955 } } peers { permanent_uuid: "7fc9d8201f4c4142866c18fbbe09a108" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39083 } } peers { permanent_uuid: "0d9d85d8ade240b9b5922ef108652037" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 42337 } }
I20260131 03:13:43.710886 6180 catalog_manager.cc:5654] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fc9d8201f4c4142866c18fbbe09a108 (127.19.84.2). New cstate: current_term: 1 leader_uuid: "7fc9d8201f4c4142866c18fbbe09a108" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44cab2527b774847b56621d1879e370a" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fc9d8201f4c4142866c18fbbe09a108" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 39083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d9d85d8ade240b9b5922ef108652037" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 42337 } health_report { overall_health: UNKNOWN } } }
I20260131 03:13:43.766749 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:43.764590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35314 (local address 127.19.84.2:39083)
0131 03:13:43.764650 (+ 60us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:43.764652 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:43.764661 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:13:43.764720 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:43.764721 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:43.764737 (+ 16us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:43.764780 (+ 43us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:13:43.765039 (+ 259us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.765536 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:43.766068 (+ 532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:43.766177 (+ 109us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:43.766316 (+ 139us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:13:43.766397 (+ 81us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:13:43.766459 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:43.766633 (+ 174us) server_negotiation.cc:300] Negotiation successful
0131 03:13:43.766648 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260131 03:13:43.768513 6323 raft_consensus.cc:1275] T 59c5c07caf4d44d492e10e084a194d95 P 44cab2527b774847b56621d1879e370a [term 1 FOLLOWER]: Refusing update from remote peer 7fc9d8201f4c4142866c18fbbe09a108: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:13:43.768513 6585 raft_consensus.cc:1275] T 59c5c07caf4d44d492e10e084a194d95 P 0d9d85d8ade240b9b5922ef108652037 [term 1 FOLLOWER]: Refusing update from remote peer 7fc9d8201f4c4142866c18fbbe09a108: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:13:43.768844 6653 consensus_queue.cc:1048] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9d85d8ade240b9b5922ef108652037" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 42337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:13:43.768972 6655 consensus_queue.cc:1048] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44cab2527b774847b56621d1879e370a" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:13:43.771860 6658 mvcc.cc:204] Tried to move back new op lower bound from 7249220500552740864 to 7249220500317167616. Current Snapshot: MvccSnapshot[applied={T|T < 7249220500552740864}]
I20260131 03:13:43.772329 6659 mvcc.cc:204] Tried to move back new op lower bound from 7249220500552740864 to 7249220500317167616. Current Snapshot: MvccSnapshot[applied={T|T < 7249220500552740864}]
I20260131 03:13:43.773830 6180 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:13:43.773926 6180 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:13:43.774708 6180 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 455e37b6c9bc4a798367852c604e3ce6: Sending DeleteTablet for 3 replicas of tablet 59c5c07caf4d44d492e10e084a194d95
I20260131 03:13:43.775065 6303 tablet_service.cc:1552] Processing DeleteTablet for tablet 59c5c07caf4d44d492e10e084a194d95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:13:43 UTC) from {username='slave'} at 127.0.0.1:39624
I20260131 03:13:43.775080 6565 tablet_service.cc:1552] Processing DeleteTablet for tablet 59c5c07caf4d44d492e10e084a194d95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:13:43 UTC) from {username='slave'} at 127.0.0.1:53836
I20260131 03:13:43.775341 6668 tablet_replica.cc:333] T 59c5c07caf4d44d492e10e084a194d95 P 44cab2527b774847b56621d1879e370a: stopping tablet replica
I20260131 03:13:43.775404 6434 tablet_service.cc:1552] Processing DeleteTablet for tablet 59c5c07caf4d44d492e10e084a194d95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:13:43 UTC) from {username='slave'} at 127.0.0.1:35302
I20260131 03:13:43.775444 6668 raft_consensus.cc:2243] T 59c5c07caf4d44d492e10e084a194d95 P 44cab2527b774847b56621d1879e370a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:13:43.775449 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 6241
I20260131 03:13:43.775544 6668 raft_consensus.cc:2272] T 59c5c07caf4d44d492e10e084a194d95 P 44cab2527b774847b56621d1879e370a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:13:43.776007 6669 tablet_replica.cc:333] T 59c5c07caf4d44d492e10e084a194d95 P 0d9d85d8ade240b9b5922ef108652037: stopping tablet replica
I20260131 03:13:43.776134 6669 raft_consensus.cc:2243] T 59c5c07caf4d44d492e10e084a194d95 P 0d9d85d8ade240b9b5922ef108652037 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:13:43.776263 6669 raft_consensus.cc:2272] T 59c5c07caf4d44d492e10e084a194d95 P 0d9d85d8ade240b9b5922ef108652037 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:13:43.776654 6669 ts_tablet_manager.cc:1916] T 59c5c07caf4d44d492e10e084a194d95 P 0d9d85d8ade240b9b5922ef108652037: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:13:43.778083 6669 ts_tablet_manager.cc:1929] T 59c5c07caf4d44d492e10e084a194d95 P 0d9d85d8ade240b9b5922ef108652037: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:13:43.778179 6669 log.cc:1199] T 59c5c07caf4d44d492e10e084a194d95 P 0d9d85d8ade240b9b5922ef108652037: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/59c5c07caf4d44d492e10e084a194d95
I20260131 03:13:43.778460 6669 ts_tablet_manager.cc:1950] T 59c5c07caf4d44d492e10e084a194d95 P 0d9d85d8ade240b9b5922ef108652037: Deleting consensus metadata
I20260131 03:13:43.779210 6166 catalog_manager.cc:4985] TS 0d9d85d8ade240b9b5922ef108652037 (127.19.84.3:42337): tablet 59c5c07caf4d44d492e10e084a194d95 (table test-table [id=d41c7bec54994f8c897ee5062c5aebb7]) successfully deleted
W20260131 03:13:43.782935 6168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.84.1:39955 (error 108)
I20260131 03:13:43.783169 6670 tablet_replica.cc:333] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108: stopping tablet replica
I20260131 03:13:43.783268 6670 raft_consensus.cc:2243] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108 [term 1 LEADER]: Raft consensus shutting down.
I20260131 03:13:43.783411 6670 raft_consensus.cc:2272] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:13:43.783453 6640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:43.783288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:39955 (local address 127.0.0.1:39632)
0131 03:13:43.783348 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:43.783418 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.1:39955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260131 03:13:43.783690 6168 catalog_manager.cc:4712] TS 44cab2527b774847b56621d1879e370a (127.19.84.1:39955): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 59c5c07caf4d44d492e10e084a194d95: Network error: Client connection negotiation failed: client connection to 127.19.84.1:39955: connect: Connection refused (error 111)
I20260131 03:13:43.783869 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 6372
I20260131 03:13:43.783953 6670 ts_tablet_manager.cc:1916] T 59c5c07caf4d44d492e10e084a194d95 P 7fc9d8201f4c4142866c18fbbe09a108: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:13:43.788898 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 6503
I20260131 03:13:43.789004 6640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:43.788846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:39083 (local address 127.0.0.1:35330)
0131 03:13:43.788919 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:43.788961 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.2:39083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260131 03:13:43.793460 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 6150
2026-01-31T03:13:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (975 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:13:43 dist-test-slave-5nnp krb5kdc[6674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:13:43 dist-test-slave-5nnp krb5kdc[6674](info): set up 2 sockets
Jan 31 03:13:43 dist-test-slave-5nnp krb5kdc[6674](info): commencing operation
krb5kdc: starting...
W20260131 03:13:45.791867 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.978s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:13:45 dist-test-slave-5nnp krb5kdc[6674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:13:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:13:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:13:45.932408 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:34611
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:34611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:13:46.012234 6690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:46.012480 6690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:46.012524 6690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:46.013978 6690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:13:46.014047 6690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:46.014070 6690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:46.014091 6690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:13:46.014132 6690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:13:46.015837 6690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:34611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:34611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:46.016091 6690 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:46.016304 6690 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:46.018901 6695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:46.018926 6696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:46.019169 6690 server_base.cc:1060] running on GCE node
W20260131 03:13:46.019171 6698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:46.019537 6690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:46.019843 6690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:46.021016 6690 hybrid_clock.cc:648] HybridClock initialized: now 1769829226020994 us; error 38 us; skew 500 ppm
Jan 31 03:13:46 dist-test-slave-5nnp krb5kdc[6674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829226, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:13:46.023337 6690 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:13:46.023960 6690 webserver.cc:492] Webserver started at http://127.19.84.62:38579/ using document root <none> and password file <none>
I20260131 03:13:46.024186 6690 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:46.024225 6690 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:46.024341 6690 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:46.025343 6690 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "90b45e1e20ba4657ba6b9d14b33bb22b"
format_stamp: "Formatted at 2026-01-31 03:13:46 on dist-test-slave-5nnp"
server_key: "a7bd47a744aded780d2a33f3bd544af9"
server_key_iv: "f19af28f1f33092776f1e8235c65743c"
server_key_version: "encryptionkey@0"
I20260131 03:13:46.025730 6690 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "90b45e1e20ba4657ba6b9d14b33bb22b"
format_stamp: "Formatted at 2026-01-31 03:13:46 on dist-test-slave-5nnp"
server_key: "a7bd47a744aded780d2a33f3bd544af9"
server_key_iv: "f19af28f1f33092776f1e8235c65743c"
server_key_version: "encryptionkey@0"
I20260131 03:13:46.027320 6690 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260131 03:13:46.028151 6705 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:46.028383 6690 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:13:46.028482 6690 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "90b45e1e20ba4657ba6b9d14b33bb22b"
format_stamp: "Formatted at 2026-01-31 03:13:46 on dist-test-slave-5nnp"
server_key: "a7bd47a744aded780d2a33f3bd544af9"
server_key_iv: "f19af28f1f33092776f1e8235c65743c"
server_key_version: "encryptionkey@0"
I20260131 03:13:46.028548 6690 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:46.042865 6690 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:46.044538 6690 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:46.044688 6690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:46.048950 6690 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:34611
I20260131 03:13:46.049026 6757 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:34611 every 8 connection(s)
I20260131 03:13:46.049350 6690 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:13:46.050011 6758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:46.052583 6758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90b45e1e20ba4657ba6b9d14b33bb22b: Bootstrap starting.
I20260131 03:13:46.053115 6758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90b45e1e20ba4657ba6b9d14b33bb22b: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:46.053324 6758 log.cc:826] T 00000000000000000000000000000000 P 90b45e1e20ba4657ba6b9d14b33bb22b: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:46.053999 6758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90b45e1e20ba4657ba6b9d14b33bb22b: No bootstrap required, opened a new log
I20260131 03:13:46.055160 6758 raft_consensus.cc:359] T 00000000000000000000000000000000 P 90b45e1e20ba4657ba6b9d14b33bb22b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b45e1e20ba4657ba6b9d14b33bb22b" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34611 } }
I20260131 03:13:46.055307 6758 raft_consensus.cc:385] T 00000000000000000000000000000000 P 90b45e1e20ba4657ba6b9d14b33bb22b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:46.055328 6758 raft_consensus.cc:740] T 00000000000000000000000000000000 P 90b45e1e20ba4657ba6b9d14b33bb22b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b45e1e20ba4657ba6b9d14b33bb22b, State: Initialized, Role: FOLLOWER
I20260131 03:13:46.055503 6758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90b45e1e20ba4657ba6b9d14b33bb22b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b45e1e20ba4657ba6b9d14b33bb22b" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34611 } }
I20260131 03:13:46.055584 6758 raft_consensus.cc:399] T 00000000000000000000000000000000 P 90b45e1e20ba4657ba6b9d14b33bb22b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:13:46.055609 6758 raft_consensus.cc:493] T 00000000000000000000000000000000 P 90b45e1e20ba4657ba6b9d14b33bb22b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:13:46.055637 6758 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 90b45e1e20ba4657ba6b9d14b33bb22b [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:46.056141 6758 raft_consensus.cc:515] T 00000000000000000000000000000000 P 90b45e1e20ba4657ba6b9d14b33bb22b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b45e1e20ba4657ba6b9d14b33bb22b" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34611 } }
I20260131 03:13:46.056229 6758 leader_election.cc:304] T 00000000000000000000000000000000 P 90b45e1e20ba4657ba6b9d14b33bb22b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90b45e1e20ba4657ba6b9d14b33bb22b; no voters:
I20260131 03:13:46.056352 6758 leader_election.cc:290] T 00000000000000000000000000000000 P 90b45e1e20ba4657ba6b9d14b33bb22b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:13:46.056479 6761 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 90b45e1e20ba4657ba6b9d14b33bb22b [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:13:46.056644 6758 sys_catalog.cc:565] T 00000000000000000000000000000000 P 90b45e1e20ba4657ba6b9d14b33bb22b [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:13:46.056632 6761 raft_consensus.cc:697] T 00000000000000000000000000000000 P 90b45e1e20ba4657ba6b9d14b33bb22b [term 1 LEADER]: Becoming Leader. State: Replica: 90b45e1e20ba4657ba6b9d14b33bb22b, State: Running, Role: LEADER
I20260131 03:13:46.056886 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 6690
I20260131 03:13:46.057027 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:13:46.056963 6761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90b45e1e20ba4657ba6b9d14b33bb22b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b45e1e20ba4657ba6b9d14b33bb22b" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34611 } }
I20260131 03:13:46.057196 19792 external_mini_cluster.cc:1468] Setting key 8d976d8d6e87c752270019d9977e60d3
I20260131 03:13:46.057391 6763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90b45e1e20ba4657ba6b9d14b33bb22b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90b45e1e20ba4657ba6b9d14b33bb22b. Latest consensus state: current_term: 1 leader_uuid: "90b45e1e20ba4657ba6b9d14b33bb22b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b45e1e20ba4657ba6b9d14b33bb22b" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34611 } } }
I20260131 03:13:46.057471 6763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90b45e1e20ba4657ba6b9d14b33bb22b [sys.catalog]: This master's current role is: LEADER
I20260131 03:13:46.057626 6762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90b45e1e20ba4657ba6b9d14b33bb22b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90b45e1e20ba4657ba6b9d14b33bb22b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b45e1e20ba4657ba6b9d14b33bb22b" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 34611 } } }
I20260131 03:13:46.057677 6762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90b45e1e20ba4657ba6b9d14b33bb22b [sys.catalog]: This master's current role is: LEADER
I20260131 03:13:46.058225 6770 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:13:46.058992 6770 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:13:46.060774 6770 catalog_manager.cc:1357] Generated new cluster ID: e7d26c18f9354393bd1a5e08646c776c
I20260131 03:13:46.060840 6770 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:13:46.066268 6770 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:13:46.067019 6770 catalog_manager.cc:1514] Loading token signing keys...
Jan 31 03:13:46 dist-test-slave-5nnp krb5kdc[6674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:13:46.072191 6772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:46.057798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57132 (local address 127.19.84.62:34611)
0131 03:13:46.058384 (+ 586us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:46.058395 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:46.058420 (+ 25us) server_negotiation.cc:408] Connection header received
0131 03:13:46.058530 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:46.058536 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:46.058590 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:46.058728 (+ 138us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:46.059735 (+ 1007us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.060664 (+ 929us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:46.061457 (+ 793us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.061683 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:46.068192 (+ 6509us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:46.068199 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:46.068199 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:46.068228 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:46.070720 (+ 2492us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:46.071096 (+ 376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:46.071100 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:46.071100 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:46.071157 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:46.071327 (+ 170us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:46.071328 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:46.071328 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:46.071586 (+ 258us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:46.071758 (+ 172us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:46.071860 (+ 102us) server_negotiation.cc:300] Negotiation successful
0131 03:13:46.072013 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":64,"threads_started":1}
I20260131 03:13:46.078311 6770 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 90b45e1e20ba4657ba6b9d14b33bb22b: Generated new TSK 0
I20260131 03:13:46.078518 6770 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:13:46.130887 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:34611
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:13:46.209903 6786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:46.210100 6786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:46.210136 6786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:46.211632 6786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:46.211692 6786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:46.211738 6786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:13:46.213335 6786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:34611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:46.213645 6786 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:46.213837 6786 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:46.216341 6794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:46.216336 6792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:46.216463 6791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:46.216686 6786 server_base.cc:1060] running on GCE node
I20260131 03:13:46.216868 6786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:46.217096 6786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:46.218248 6786 hybrid_clock.cc:648] HybridClock initialized: now 1769829226218208 us; error 54 us; skew 500 ppm
Jan 31 03:13:46 dist-test-slave-5nnp krb5kdc[6674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829226, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:13:46.220587 6786 init.cc:377] Logged in from keytab as kudu/127.19.84.1@KRBTEST.COM (short username kudu)
I20260131 03:13:46.221131 6786 webserver.cc:492] Webserver started at http://127.19.84.1:39283/ using document root <none> and password file <none>
I20260131 03:13:46.221334 6786 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:46.221390 6786 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:46.221510 6786 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:46.222437 6786 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "3fa8d015670d4fb398e6d6bdc0b45e4e"
format_stamp: "Formatted at 2026-01-31 03:13:46 on dist-test-slave-5nnp"
server_key: "e11fce302e2b428f7e9d48091c872bd4"
server_key_iv: "76b59614b432f4d09575379d3f5f3bbd"
server_key_version: "encryptionkey@0"
I20260131 03:13:46.222759 6786 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "3fa8d015670d4fb398e6d6bdc0b45e4e"
format_stamp: "Formatted at 2026-01-31 03:13:46 on dist-test-slave-5nnp"
server_key: "e11fce302e2b428f7e9d48091c872bd4"
server_key_iv: "76b59614b432f4d09575379d3f5f3bbd"
server_key_version: "encryptionkey@0"
I20260131 03:13:46.224048 6786 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:13:46.224792 6802 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:46.224965 6786 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260131 03:13:46.225080 6786 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "3fa8d015670d4fb398e6d6bdc0b45e4e"
format_stamp: "Formatted at 2026-01-31 03:13:46 on dist-test-slave-5nnp"
server_key: "e11fce302e2b428f7e9d48091c872bd4"
server_key_iv: "76b59614b432f4d09575379d3f5f3bbd"
server_key_version: "encryptionkey@0"
I20260131 03:13:46.225158 6786 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:46.246276 6786 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:46.248459 6786 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:46.248651 6786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:46.248910 6786 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:13:46.249264 6786 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:13:46.249321 6786 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:46.249365 6786 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:13:46.249395 6786 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:46.256086 6786 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:35485
I20260131 03:13:46.256122 6915 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:35485 every 8 connection(s)
I20260131 03:13:46.256423 6786 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
Jan 31 03:13:46 dist-test-slave-5nnp krb5kdc[6674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829226, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.1@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:13:46.265134 6772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:46.257138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:34607 (local address 127.19.84.62:34611)
0131 03:13:46.257313 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:46.257315 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:46.257358 (+ 43us) server_negotiation.cc:408] Connection header received
0131 03:13:46.257848 (+ 490us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:46.257850 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:46.257858 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:46.257921 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:46.258708 (+ 787us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.259208 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:46.259846 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.259959 (+ 113us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:46.262257 (+ 2298us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:46.262262 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:46.262263 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:46.262273 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:46.263777 (+ 1504us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:46.264219 (+ 442us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:46.264220 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:46.264221 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:46.264250 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:46.264475 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:46.264476 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:46.264476 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:46.264575 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:46.264651 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:46.264929 (+ 278us) server_negotiation.cc:300] Negotiation successful
0131 03:13:46.265026 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260131 03:13:46.265254 6918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:46.257043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34611 (local address 127.19.84.1:34607)
0131 03:13:46.257241 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:46.257278 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:46.257770 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:46.257988 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:46.257991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:46.258347 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:13:46.258630 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:46.258634 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.259313 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:46.259314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:46.259772 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:46.259774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.259886 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:46.260744 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:13:46.260761 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:13:46.262151 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:13:46.263899 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:46.263906 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:46.263909 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:46.264130 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:46.264328 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:46.264329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:46.264329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:46.264407 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:46.264640 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:13:46.264646 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:13:46.264777 (+ 131us) client_negotiation.cc:769] Sending connection context
0131 03:13:46.264910 (+ 133us) client_negotiation.cc:240] Negotiation successful
0131 03:13:46.265058 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260131 03:13:46.265789 6916 heartbeater.cc:344] Connected to a master server at 127.19.84.62:34611
I20260131 03:13:46.265877 6916 heartbeater.cc:461] Registering TS with master...
I20260131 03:13:46.265933 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 6786
I20260131 03:13:46.266050 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:13:46.266155 6916 heartbeater.cc:507] Master 127.19.84.62:34611 requested a full tablet report, sending...
I20260131 03:13:46.266321 19792 external_mini_cluster.cc:1468] Setting key cb35e41a040168a554b7622336ad01fe
I20260131 03:13:46.266673 6722 ts_manager.cc:194] Registered new tserver with Master: 3fa8d015670d4fb398e6d6bdc0b45e4e (127.19.84.1:35485)
I20260131 03:13:46.267329 6722 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.1@KRBTEST.COM'} at 127.19.84.1:34607
WARNING: no policy specified for kudu/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:13:46.315618 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:34611
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:13:46.400537 6923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:46.400749 6923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:46.400784 6923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:46.402302 6923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:46.402379 6923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:46.402423 6923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:13:46.404134 6923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:34611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:46.404407 6923 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:46.404606 6923 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:46.407428 6928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:46.407447 6929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:46.407582 6923 server_base.cc:1060] running on GCE node
W20260131 03:13:46.407428 6931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:46.407800 6923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:46.407976 6923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:46.409112 6923 hybrid_clock.cc:648] HybridClock initialized: now 1769829226409099 us; error 52 us; skew 500 ppm
Jan 31 03:13:46 dist-test-slave-5nnp krb5kdc[6674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829226, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:13:46.411532 6923 init.cc:377] Logged in from keytab as kudu/127.19.84.2@KRBTEST.COM (short username kudu)
I20260131 03:13:46.412014 6923 webserver.cc:492] Webserver started at http://127.19.84.2:45299/ using document root <none> and password file <none>
I20260131 03:13:46.412227 6923 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:46.412286 6923 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:46.412407 6923 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:46.413369 6923 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "77302c2270ca45e3b6e1bef96e62ac96"
format_stamp: "Formatted at 2026-01-31 03:13:46 on dist-test-slave-5nnp"
server_key: "23b361d9ad9776294cd2f705b48c16e9"
server_key_iv: "19dd71ab258bf127abcd2f96d0274bdd"
server_key_version: "encryptionkey@0"
I20260131 03:13:46.413697 6923 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "77302c2270ca45e3b6e1bef96e62ac96"
format_stamp: "Formatted at 2026-01-31 03:13:46 on dist-test-slave-5nnp"
server_key: "23b361d9ad9776294cd2f705b48c16e9"
server_key_iv: "19dd71ab258bf127abcd2f96d0274bdd"
server_key_version: "encryptionkey@0"
I20260131 03:13:46.414920 6923 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:13:46.415818 6938 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:46.416074 6923 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260131 03:13:46.416154 6923 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "77302c2270ca45e3b6e1bef96e62ac96"
format_stamp: "Formatted at 2026-01-31 03:13:46 on dist-test-slave-5nnp"
server_key: "23b361d9ad9776294cd2f705b48c16e9"
server_key_iv: "19dd71ab258bf127abcd2f96d0274bdd"
server_key_version: "encryptionkey@0"
I20260131 03:13:46.416205 6923 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:46.425612 6923 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:46.427672 6923 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:46.427783 6923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:46.428015 6923 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:13:46.428287 6923 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:13:46.428318 6923 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:46.428340 6923 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:13:46.428352 6923 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:46.434599 6923 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:45217
I20260131 03:13:46.434670 7051 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:45217 every 8 connection(s)
I20260131 03:13:46.435003 6923 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
I20260131 03:13:46.440711 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 6923
I20260131 03:13:46.440812 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
Jan 31 03:13:46 dist-test-slave-5nnp krb5kdc[6674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829226, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.2@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:13:46.441010 19792 external_mini_cluster.cc:1468] Setting key 09994bf387bd5c0366f8dd2f9ea63cc3
I20260131 03:13:46.444654 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:46.435762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34611 (local address 127.19.84.2:52631)
0131 03:13:46.435937 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:46.435966 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:46.436467 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:46.436739 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:46.436743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:46.437078 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:13:46.437373 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:46.437377 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.438191 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:46.438192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:46.438628 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:46.438630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.438746 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:46.439765 (+ 1019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:13:46.439784 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:13:46.441363 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:13:46.443334 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:46.443340 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:46.443343 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:46.443630 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:46.443835 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:46.443835 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:46.443836 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:46.443909 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:46.444181 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:13:46.444185 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:13:46.444306 (+ 121us) client_negotiation.cc:769] Sending connection context
0131 03:13:46.444404 (+ 98us) client_negotiation.cc:240] Negotiation successful
0131 03:13:46.444511 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
I20260131 03:13:46.444706 6772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:46.435794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:52631 (local address 127.19.84.62:34611)
0131 03:13:46.435870 (+ 76us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:46.435873 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:46.436086 (+ 213us) server_negotiation.cc:408] Connection header received
0131 03:13:46.436547 (+ 461us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:46.436549 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:46.436559 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:46.436616 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:46.437495 (+ 879us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.438088 (+ 593us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:46.438745 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.438883 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:46.441467 (+ 2584us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:46.441472 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:46.441472 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:46.441481 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:46.443138 (+ 1657us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:46.443720 (+ 582us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:46.443722 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:46.443722 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:46.443755 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:46.443976 (+ 221us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:46.443978 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:46.443978 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:46.444114 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:46.444186 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:46.444450 (+ 264us) server_negotiation.cc:300] Negotiation successful
0131 03:13:46.444565 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260131 03:13:46.445271 7052 heartbeater.cc:344] Connected to a master server at 127.19.84.62:34611
I20260131 03:13:46.445360 7052 heartbeater.cc:461] Registering TS with master...
I20260131 03:13:46.445574 7052 heartbeater.cc:507] Master 127.19.84.62:34611 requested a full tablet report, sending...
I20260131 03:13:46.446025 6722 ts_manager.cc:194] Registered new tserver with Master: 77302c2270ca45e3b6e1bef96e62ac96 (127.19.84.2:45217)
I20260131 03:13:46.446437 6722 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.2@KRBTEST.COM'} at 127.19.84.2:52631
WARNING: no policy specified for kudu/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:13:46.490875 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:34611
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:13:46.572214 7059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:46.572410 7059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:46.572435 7059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:46.573930 7059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:46.573989 7059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:46.574018 7059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:13:46.575752 7059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.19.84.62:34611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:46.576018 7059 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:46.576220 7059 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:46.579149 7067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:46.579257 7065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:46.579165 7064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:46.579540 7059 server_base.cc:1060] running on GCE node
I20260131 03:13:46.579710 7059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:46.579928 7059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:46.581085 7059 hybrid_clock.cc:648] HybridClock initialized: now 1769829226581071 us; error 38 us; skew 500 ppm
Jan 31 03:13:46 dist-test-slave-5nnp krb5kdc[6674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829226, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:13:46.583518 7059 init.cc:377] Logged in from keytab as kudu/127.19.84.3@KRBTEST.COM (short username kudu)
I20260131 03:13:46.584041 7059 webserver.cc:492] Webserver started at http://127.19.84.3:39775/ using document root <none> and password file <none>
I20260131 03:13:46.584278 7059 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:46.584339 7059 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:46.584460 7059 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:46.585434 7059 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "ed96dc32101246e3b98f2fd4cb54dfa7"
format_stamp: "Formatted at 2026-01-31 03:13:46 on dist-test-slave-5nnp"
server_key: "55bed59cba275c2678da7c9dbcb14192"
server_key_iv: "ee3b7909e6930d7843364b4ba1347434"
server_key_version: "encryptionkey@0"
I20260131 03:13:46.585750 7059 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "ed96dc32101246e3b98f2fd4cb54dfa7"
format_stamp: "Formatted at 2026-01-31 03:13:46 on dist-test-slave-5nnp"
server_key: "55bed59cba275c2678da7c9dbcb14192"
server_key_iv: "ee3b7909e6930d7843364b4ba1347434"
server_key_version: "encryptionkey@0"
I20260131 03:13:46.586944 7059 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:13:46.587742 7074 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:46.587941 7059 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:13:46.588027 7059 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "ed96dc32101246e3b98f2fd4cb54dfa7"
format_stamp: "Formatted at 2026-01-31 03:13:46 on dist-test-slave-5nnp"
server_key: "55bed59cba275c2678da7c9dbcb14192"
server_key_iv: "ee3b7909e6930d7843364b4ba1347434"
server_key_version: "encryptionkey@0"
I20260131 03:13:46.588106 7059 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:46.597756 7059 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:46.599822 7059 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:46.599984 7059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:46.600258 7059 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:13:46.600796 7059 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:13:46.600857 7059 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:46.600901 7059 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:13:46.600930 7059 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:46.607489 7059 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:36467
I20260131 03:13:46.607527 7188 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:36467 every 8 connection(s)
I20260131 03:13:46.607834 7059 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
Jan 31 03:13:46 dist-test-slave-5nnp krb5kdc[6674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829226, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.3@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:13:46.615962 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 7059
I20260131 03:13:46.616077 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:13:46.616281 19792 external_mini_cluster.cc:1468] Setting key 7f94ffb6900d760c52f056b7969b6bb8
I20260131 03:13:46.616780 6772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:46.608555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:52337 (local address 127.19.84.62:34611)
0131 03:13:46.608633 (+ 78us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:46.608635 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:46.608803 (+ 168us) server_negotiation.cc:408] Connection header received
0131 03:13:46.609345 (+ 542us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:46.609347 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:46.609359 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:46.609421 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:46.610159 (+ 738us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.610646 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:46.611200 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.611366 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:46.613701 (+ 2335us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:46.613706 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:46.613706 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:46.613714 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:46.615128 (+ 1414us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:46.615696 (+ 568us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:46.615697 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:46.615697 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:46.615720 (+ 23us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:46.615986 (+ 266us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:46.615988 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:46.615988 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:46.616138 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:46.616209 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:46.616552 (+ 343us) server_negotiation.cc:300] Negotiation successful
0131 03:13:46.616651 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260131 03:13:46.616930 7191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:46.608526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:34611 (local address 127.19.84.3:52337)
0131 03:13:46.608683 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:46.608722 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:46.609224 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:46.609480 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:46.609488 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:46.609786 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:13:46.610058 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:46.610061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.610736 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:46.610738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:46.611112 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:46.611114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.611262 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:46.612191 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:13:46.612207 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:13:46.613594 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:13:46.615232 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:46.615239 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:46.615241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:46.615624 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:46.615801 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:13:46.615802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:13:46.615802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:13:46.615887 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:13:46.616197 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:13:46.616202 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:13:46.616351 (+ 149us) client_negotiation.cc:769] Sending connection context
0131 03:13:46.616514 (+ 163us) client_negotiation.cc:240] Negotiation successful
0131 03:13:46.616687 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260131 03:13:46.617484 7189 heartbeater.cc:344] Connected to a master server at 127.19.84.62:34611
I20260131 03:13:46.617620 7189 heartbeater.cc:461] Registering TS with master...
I20260131 03:13:46.617837 7189 heartbeater.cc:507] Master 127.19.84.62:34611 requested a full tablet report, sending...
I20260131 03:13:46.618259 6722 ts_manager.cc:194] Registered new tserver with Master: ed96dc32101246e3b98f2fd4cb54dfa7 (127.19.84.3:36467)
I20260131 03:13:46.618649 6722 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.84.3@KRBTEST.COM'} at 127.19.84.3:52337
I20260131 03:13:46.628530 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:13:46.635838 6772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:46.629954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57142 (local address 127.19.84.62:34611)
0131 03:13:46.630044 (+ 90us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:46.630046 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:46.630189 (+ 143us) server_negotiation.cc:408] Connection header received
0131 03:13:46.630215 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:46.630216 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:46.630226 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:46.630286 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:46.631027 (+ 741us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.631602 (+ 575us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:46.632206 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.632369 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:46.633393 (+ 1024us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:46.633396 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:46.633396 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:46.633402 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:46.634822 (+ 1420us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:46.635160 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:46.635162 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:46.635162 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:46.635183 (+ 21us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:46.635315 (+ 132us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:46.635316 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:46.635316 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:46.635481 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:46.635560 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:46.635696 (+ 136us) server_negotiation.cc:300] Negotiation successful
0131 03:13:46.635768 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260131 03:13:46.637360 6722 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:13:46.638590 6722 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:13:46.644634 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:46.640933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:35485 (local address 127.0.0.1:43000)
0131 03:13:46.641371 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:46.641381 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:46.641461 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:46.641742 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:46.641746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:46.641748 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:46.642032 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:46.642034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.642921 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:46.642922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:46.643671 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:46.643672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.644405 (+ 733us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:46.644419 (+ 14us) client_negotiation.cc:769] Sending connection context
0131 03:13:46.644487 (+ 68us) client_negotiation.cc:240] Negotiation successful
0131 03:13:46.644519 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":4,"thread_start_us":75,"threads_started":1}
I20260131 03:13:46.644654 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:46.641120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43000 (local address 127.19.84.1:35485)
0131 03:13:46.641284 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:46.641292 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:46.641453 (+ 161us) server_negotiation.cc:408] Connection header received
0131 03:13:46.641544 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:46.641547 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:46.641568 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:46.641631 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:46.642136 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.642844 (+ 708us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:46.643793 (+ 949us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.644320 (+ 527us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:46.644379 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:46.644524 (+ 145us) server_negotiation.cc:300] Negotiation successful
0131 03:13:46.644578 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260131 03:13:46.645135 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:46.641186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44374 (local address 127.19.84.2:45217)
0131 03:13:46.641490 (+ 304us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:46.641497 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:46.641666 (+ 169us) server_negotiation.cc:408] Connection header received
0131 03:13:46.641708 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:46.641711 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:46.641728 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:46.641788 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:46.642172 (+ 384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.643304 (+ 1132us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:46.644412 (+ 1108us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.644944 (+ 532us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:46.644989 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:46.645018 (+ 29us) server_negotiation.cc:300] Negotiation successful
0131 03:13:46.645059 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":76,"threads_started":1}
I20260131 03:13:46.645517 6850 tablet_service.cc:1505] Processing CreateTablet for tablet 75a95ee19134477191da3b0065aafadf (DEFAULT_TABLE table=test-table [id=56ee6cff1e7a489783f63d094ceb78e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:13:46.645838 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:46.640933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:36467 (local address 127.0.0.1:46114)
0131 03:13:46.641310 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:46.641343 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:46.641424 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:46.641762 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:46.641763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:46.641765 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:46.642029 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:46.642031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.644666 (+ 2635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:46.644667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:46.645608 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:46.645611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.645692 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:46.645696 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:13:46.645726 (+ 30us) client_negotiation.cc:240] Negotiation successful
0131 03:13:46.645767 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":89,"threads_started":1}
I20260131 03:13:46.645846 6850 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75a95ee19134477191da3b0065aafadf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:46.644634 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:46.640956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:45217 (local address 127.0.0.1:44374)
0131 03:13:46.641548 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:46.641556 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:46.641606 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:46.641868 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:46.641869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:46.641870 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:46.642094 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:46.642096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.643411 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:46.643412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:46.644299 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:46.644302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.644389 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:46.644419 (+ 30us) client_negotiation.cc:769] Sending connection context
0131 03:13:46.644471 (+ 52us) client_negotiation.cc:240] Negotiation successful
0131 03:13:46.644517 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":93,"threads_started":1}
I20260131 03:13:46.646654 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:46.641209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46114 (local address 127.19.84.3:36467)
0131 03:13:46.641382 (+ 173us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:46.641389 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:46.641400 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:13:46.641530 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:46.641533 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:46.641594 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:46.641643 (+ 49us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:46.642108 (+ 465us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.643300 (+ 1192us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:46.645718 (+ 2418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.646424 (+ 706us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:46.646486 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:46.646520 (+ 34us) server_negotiation.cc:300] Negotiation successful
0131 03:13:46.646572 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
I20260131 03:13:46.646672 6986 tablet_service.cc:1505] Processing CreateTablet for tablet 75a95ee19134477191da3b0065aafadf (DEFAULT_TABLE table=test-table [id=56ee6cff1e7a489783f63d094ceb78e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:13:46.646947 6986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75a95ee19134477191da3b0065aafadf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:46.647183 7123 tablet_service.cc:1505] Processing CreateTablet for tablet 75a95ee19134477191da3b0065aafadf (DEFAULT_TABLE table=test-table [id=56ee6cff1e7a489783f63d094ceb78e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:13:46.647486 7123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75a95ee19134477191da3b0065aafadf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:46.648514 7206 tablet_bootstrap.cc:492] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e: Bootstrap starting.
I20260131 03:13:46.649041 7206 tablet_bootstrap.cc:654] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:46.649341 7206 log.cc:826] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:46.650133 7206 tablet_bootstrap.cc:492] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e: No bootstrap required, opened a new log
I20260131 03:13:46.650215 7206 ts_tablet_manager.cc:1403] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:13:46.650315 7209 tablet_bootstrap.cc:492] T 75a95ee19134477191da3b0065aafadf P ed96dc32101246e3b98f2fd4cb54dfa7: Bootstrap starting.
I20260131 03:13:46.650657 7208 tablet_bootstrap.cc:492] T 75a95ee19134477191da3b0065aafadf P 77302c2270ca45e3b6e1bef96e62ac96: Bootstrap starting.
I20260131 03:13:46.650997 7209 tablet_bootstrap.cc:654] T 75a95ee19134477191da3b0065aafadf P ed96dc32101246e3b98f2fd4cb54dfa7: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:46.651335 7208 tablet_bootstrap.cc:654] T 75a95ee19134477191da3b0065aafadf P 77302c2270ca45e3b6e1bef96e62ac96: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:46.651340 7209 log.cc:826] T 75a95ee19134477191da3b0065aafadf P ed96dc32101246e3b98f2fd4cb54dfa7: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:46.651664 7208 log.cc:826] T 75a95ee19134477191da3b0065aafadf P 77302c2270ca45e3b6e1bef96e62ac96: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:46.651731 7206 raft_consensus.cc:359] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed96dc32101246e3b98f2fd4cb54dfa7" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36467 } } peers { permanent_uuid: "3fa8d015670d4fb398e6d6bdc0b45e4e" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35485 } } peers { permanent_uuid: "77302c2270ca45e3b6e1bef96e62ac96" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45217 } }
I20260131 03:13:46.651919 7206 raft_consensus.cc:385] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:46.651988 7206 raft_consensus.cc:740] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fa8d015670d4fb398e6d6bdc0b45e4e, State: Initialized, Role: FOLLOWER
I20260131 03:13:46.652127 7206 consensus_queue.cc:260] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed96dc32101246e3b98f2fd4cb54dfa7" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36467 } } peers { permanent_uuid: "3fa8d015670d4fb398e6d6bdc0b45e4e" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35485 } } peers { permanent_uuid: "77302c2270ca45e3b6e1bef96e62ac96" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45217 } }
I20260131 03:13:46.652189 7209 tablet_bootstrap.cc:492] T 75a95ee19134477191da3b0065aafadf P ed96dc32101246e3b98f2fd4cb54dfa7: No bootstrap required, opened a new log
I20260131 03:13:46.652253 7209 ts_tablet_manager.cc:1403] T 75a95ee19134477191da3b0065aafadf P ed96dc32101246e3b98f2fd4cb54dfa7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:13:46.652407 7206 ts_tablet_manager.cc:1434] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:13:46.652427 7208 tablet_bootstrap.cc:492] T 75a95ee19134477191da3b0065aafadf P 77302c2270ca45e3b6e1bef96e62ac96: No bootstrap required, opened a new log
I20260131 03:13:46.652484 7208 ts_tablet_manager.cc:1403] T 75a95ee19134477191da3b0065aafadf P 77302c2270ca45e3b6e1bef96e62ac96: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260131 03:13:46.652550 6916 heartbeater.cc:499] Master 127.19.84.62:34611 was elected leader, sending a full tablet report...
I20260131 03:13:46.654003 7209 raft_consensus.cc:359] T 75a95ee19134477191da3b0065aafadf P ed96dc32101246e3b98f2fd4cb54dfa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed96dc32101246e3b98f2fd4cb54dfa7" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36467 } } peers { permanent_uuid: "3fa8d015670d4fb398e6d6bdc0b45e4e" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35485 } } peers { permanent_uuid: "77302c2270ca45e3b6e1bef96e62ac96" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45217 } }
I20260131 03:13:46.654168 7209 raft_consensus.cc:385] T 75a95ee19134477191da3b0065aafadf P ed96dc32101246e3b98f2fd4cb54dfa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:46.654204 7209 raft_consensus.cc:740] T 75a95ee19134477191da3b0065aafadf P ed96dc32101246e3b98f2fd4cb54dfa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed96dc32101246e3b98f2fd4cb54dfa7, State: Initialized, Role: FOLLOWER
I20260131 03:13:46.654179 7208 raft_consensus.cc:359] T 75a95ee19134477191da3b0065aafadf P 77302c2270ca45e3b6e1bef96e62ac96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed96dc32101246e3b98f2fd4cb54dfa7" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36467 } } peers { permanent_uuid: "3fa8d015670d4fb398e6d6bdc0b45e4e" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35485 } } peers { permanent_uuid: "77302c2270ca45e3b6e1bef96e62ac96" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45217 } }
I20260131 03:13:46.654297 7208 raft_consensus.cc:385] T 75a95ee19134477191da3b0065aafadf P 77302c2270ca45e3b6e1bef96e62ac96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:46.654337 7208 raft_consensus.cc:740] T 75a95ee19134477191da3b0065aafadf P 77302c2270ca45e3b6e1bef96e62ac96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77302c2270ca45e3b6e1bef96e62ac96, State: Initialized, Role: FOLLOWER
I20260131 03:13:46.654302 7209 consensus_queue.cc:260] T 75a95ee19134477191da3b0065aafadf P ed96dc32101246e3b98f2fd4cb54dfa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed96dc32101246e3b98f2fd4cb54dfa7" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36467 } } peers { permanent_uuid: "3fa8d015670d4fb398e6d6bdc0b45e4e" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35485 } } peers { permanent_uuid: "77302c2270ca45e3b6e1bef96e62ac96" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45217 } }
I20260131 03:13:46.654438 7208 consensus_queue.cc:260] T 75a95ee19134477191da3b0065aafadf P 77302c2270ca45e3b6e1bef96e62ac96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed96dc32101246e3b98f2fd4cb54dfa7" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36467 } } peers { permanent_uuid: "3fa8d015670d4fb398e6d6bdc0b45e4e" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35485 } } peers { permanent_uuid: "77302c2270ca45e3b6e1bef96e62ac96" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45217 } }
I20260131 03:13:46.654523 7209 ts_tablet_manager.cc:1434] T 75a95ee19134477191da3b0065aafadf P ed96dc32101246e3b98f2fd4cb54dfa7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:13:46.654589 7189 heartbeater.cc:499] Master 127.19.84.62:34611 was elected leader, sending a full tablet report...
I20260131 03:13:46.654690 7052 heartbeater.cc:499] Master 127.19.84.62:34611 was elected leader, sending a full tablet report...
I20260131 03:13:46.654645 7208 ts_tablet_manager.cc:1434] T 75a95ee19134477191da3b0065aafadf P 77302c2270ca45e3b6e1bef96e62ac96: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260131 03:13:46.685572 7053 tablet.cc:2378] T 75a95ee19134477191da3b0065aafadf P 77302c2270ca45e3b6e1bef96e62ac96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:13:46.687472 7212 raft_consensus.cc:493] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:13:46.687616 7212 raft_consensus.cc:515] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed96dc32101246e3b98f2fd4cb54dfa7" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36467 } } peers { permanent_uuid: "3fa8d015670d4fb398e6d6bdc0b45e4e" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35485 } } peers { permanent_uuid: "77302c2270ca45e3b6e1bef96e62ac96" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45217 } }
I20260131 03:13:46.688001 7212 leader_election.cc:290] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed96dc32101246e3b98f2fd4cb54dfa7 (127.19.84.3:36467), 77302c2270ca45e3b6e1bef96e62ac96 (127.19.84.2:45217)
I20260131 03:13:46.690850 6918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:46.688134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:36467 (local address 127.19.84.1:38633)
0131 03:13:46.688177 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:46.688189 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:46.688240 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:46.688456 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:46.688457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:46.688460 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:46.688730 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:46.688732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.689677 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:46.689678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:46.690606 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:46.690609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.690686 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:46.690691 (+ 5us) client_negotiation.cc:769] Sending connection context
0131 03:13:46.690726 (+ 35us) client_negotiation.cc:240] Negotiation successful
0131 03:13:46.690759 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260131 03:13:46.691481 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:46.688288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:50611 (local address 127.19.84.2:45217)
0131 03:13:46.688324 (+ 36us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:46.688326 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:46.688411 (+ 85us) server_negotiation.cc:408] Connection header received
0131 03:13:46.688468 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:46.688469 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:46.688483 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:46.688537 (+ 54us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:46.688884 (+ 347us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.689599 (+ 715us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:46.690813 (+ 1214us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.691306 (+ 493us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:46.691314 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:46.691337 (+ 23us) server_negotiation.cc:300] Negotiation successful
0131 03:13:46.691398 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260131 03:13:46.691499 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:46.688152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:45217 (local address 127.19.84.1:50611)
0131 03:13:46.688341 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:46.688348 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:46.688392 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:46.688556 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:46.688557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:46.688559 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:46.688806 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:46.688807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.689697 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:46.689698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:46.690618 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:46.690620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.690688 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:46.690691 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:13:46.690726 (+ 35us) client_negotiation.cc:240] Negotiation successful
0131 03:13:46.690759 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":113,"threads_started":1}
I20260131 03:13:46.691771 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:46.688202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:38633 (local address 127.19.84.3:36467)
0131 03:13:46.688250 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:46.688252 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:46.688265 (+ 13us) server_negotiation.cc:408] Connection header received
0131 03:13:46.688335 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:46.688337 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:46.688351 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:46.688435 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:46.688803 (+ 368us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.689579 (+ 776us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:46.690717 (+ 1138us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.691612 (+ 895us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:46.691622 (+ 10us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:46.691650 (+ 28us) server_negotiation.cc:300] Negotiation successful
0131 03:13:46.691684 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260131 03:13:46.691838 7006 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75a95ee19134477191da3b0065aafadf" candidate_uuid: "3fa8d015670d4fb398e6d6bdc0b45e4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77302c2270ca45e3b6e1bef96e62ac96" is_pre_election: true
I20260131 03:13:46.691980 7006 raft_consensus.cc:2468] T 75a95ee19134477191da3b0065aafadf P 77302c2270ca45e3b6e1bef96e62ac96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fa8d015670d4fb398e6d6bdc0b45e4e in term 0.
I20260131 03:13:46.692070 7143 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75a95ee19134477191da3b0065aafadf" candidate_uuid: "3fa8d015670d4fb398e6d6bdc0b45e4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed96dc32101246e3b98f2fd4cb54dfa7" is_pre_election: true
I20260131 03:13:46.692225 7143 raft_consensus.cc:2468] T 75a95ee19134477191da3b0065aafadf P ed96dc32101246e3b98f2fd4cb54dfa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fa8d015670d4fb398e6d6bdc0b45e4e in term 0.
I20260131 03:13:46.692250 6805 leader_election.cc:304] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fa8d015670d4fb398e6d6bdc0b45e4e, 77302c2270ca45e3b6e1bef96e62ac96; no voters:
I20260131 03:13:46.692395 7212 raft_consensus.cc:2804] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:13:46.692471 7212 raft_consensus.cc:493] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:13:46.692502 7212 raft_consensus.cc:3060] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:46.693130 7212 raft_consensus.cc:515] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed96dc32101246e3b98f2fd4cb54dfa7" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36467 } } peers { permanent_uuid: "3fa8d015670d4fb398e6d6bdc0b45e4e" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35485 } } peers { permanent_uuid: "77302c2270ca45e3b6e1bef96e62ac96" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45217 } }
I20260131 03:13:46.693275 7212 leader_election.cc:290] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e [CANDIDATE]: Term 1 election: Requested vote from peers ed96dc32101246e3b98f2fd4cb54dfa7 (127.19.84.3:36467), 77302c2270ca45e3b6e1bef96e62ac96 (127.19.84.2:45217)
I20260131 03:13:46.693418 7143 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75a95ee19134477191da3b0065aafadf" candidate_uuid: "3fa8d015670d4fb398e6d6bdc0b45e4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed96dc32101246e3b98f2fd4cb54dfa7"
I20260131 03:13:46.693511 7143 raft_consensus.cc:3060] T 75a95ee19134477191da3b0065aafadf P ed96dc32101246e3b98f2fd4cb54dfa7 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:46.693487 7006 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75a95ee19134477191da3b0065aafadf" candidate_uuid: "3fa8d015670d4fb398e6d6bdc0b45e4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77302c2270ca45e3b6e1bef96e62ac96"
I20260131 03:13:46.693574 7006 raft_consensus.cc:3060] T 75a95ee19134477191da3b0065aafadf P 77302c2270ca45e3b6e1bef96e62ac96 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:46.694235 7143 raft_consensus.cc:2468] T 75a95ee19134477191da3b0065aafadf P ed96dc32101246e3b98f2fd4cb54dfa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fa8d015670d4fb398e6d6bdc0b45e4e in term 1.
I20260131 03:13:46.694250 7006 raft_consensus.cc:2468] T 75a95ee19134477191da3b0065aafadf P 77302c2270ca45e3b6e1bef96e62ac96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fa8d015670d4fb398e6d6bdc0b45e4e in term 1.
I20260131 03:13:46.694416 6806 leader_election.cc:304] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fa8d015670d4fb398e6d6bdc0b45e4e, ed96dc32101246e3b98f2fd4cb54dfa7; no voters:
I20260131 03:13:46.694548 7212 raft_consensus.cc:2804] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:13:46.694697 7212 raft_consensus.cc:697] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e [term 1 LEADER]: Becoming Leader. State: Replica: 3fa8d015670d4fb398e6d6bdc0b45e4e, State: Running, Role: LEADER
I20260131 03:13:46.694794 7212 consensus_queue.cc:237] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed96dc32101246e3b98f2fd4cb54dfa7" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36467 } } peers { permanent_uuid: "3fa8d015670d4fb398e6d6bdc0b45e4e" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35485 } } peers { permanent_uuid: "77302c2270ca45e3b6e1bef96e62ac96" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45217 } }
I20260131 03:13:46.695694 6722 catalog_manager.cc:5654] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fa8d015670d4fb398e6d6bdc0b45e4e (127.19.84.1). New cstate: current_term: 1 leader_uuid: "3fa8d015670d4fb398e6d6bdc0b45e4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed96dc32101246e3b98f2fd4cb54dfa7" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fa8d015670d4fb398e6d6bdc0b45e4e" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 35485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77302c2270ca45e3b6e1bef96e62ac96" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45217 } health_report { overall_health: UNKNOWN } } }
I20260131 03:13:46.703790 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:46.701658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.19.84.1:35485)
0131 03:13:46.701728 (+ 70us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:46.701730 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:46.701739 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:13:46.701784 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:46.701784 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:46.701806 (+ 22us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:46.701866 (+ 60us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:13:46.702140 (+ 274us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.702613 (+ 473us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:46.703213 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.703405 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:46.703441 (+ 36us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:13:46.703524 (+ 83us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:13:46.703617 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:46.703679 (+ 62us) server_negotiation.cc:300] Negotiation successful
0131 03:13:46.703702 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260131 03:13:46.705451 7143 raft_consensus.cc:1275] T 75a95ee19134477191da3b0065aafadf P ed96dc32101246e3b98f2fd4cb54dfa7 [term 1 FOLLOWER]: Refusing update from remote peer 3fa8d015670d4fb398e6d6bdc0b45e4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:13:46.705451 7006 raft_consensus.cc:1275] T 75a95ee19134477191da3b0065aafadf P 77302c2270ca45e3b6e1bef96e62ac96 [term 1 FOLLOWER]: Refusing update from remote peer 3fa8d015670d4fb398e6d6bdc0b45e4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:13:46.705770 7216 consensus_queue.cc:1048] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed96dc32101246e3b98f2fd4cb54dfa7" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 36467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:13:46.705885 7212 consensus_queue.cc:1048] T 75a95ee19134477191da3b0065aafadf P 3fa8d015670d4fb398e6d6bdc0b45e4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "77302c2270ca45e3b6e1bef96e62ac96" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 45217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:13:46.708843 7221 mvcc.cc:204] Tried to move back new op lower bound from 7249220512582701056 to 7249220512542461952. Current Snapshot: MvccSnapshot[applied={T|T < 7249220512582701056}]
I20260131 03:13:46.709762 7220 mvcc.cc:204] Tried to move back new op lower bound from 7249220512582701056 to 7249220512542461952. Current Snapshot: MvccSnapshot[applied={T|T < 7249220512582701056 or (T in {7249220512582701056})}]
I20260131 03:13:46.710057 6722 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:13:46.710132 6722 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:13:46.710796 6722 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 90b45e1e20ba4657ba6b9d14b33bb22b: Sending DeleteTablet for 3 replicas of tablet 75a95ee19134477191da3b0065aafadf
I20260131 03:13:46.711126 7123 tablet_service.cc:1552] Processing DeleteTablet for tablet 75a95ee19134477191da3b0065aafadf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:13:46 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:46114
I20260131 03:13:46.711129 6986 tablet_service.cc:1552] Processing DeleteTablet for tablet 75a95ee19134477191da3b0065aafadf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:13:46 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:44374
I20260131 03:13:46.711331 6850 tablet_service.cc:1552] Processing DeleteTablet for tablet 75a95ee19134477191da3b0065aafadf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:13:46 UTC) from {username='kudu', principal='kudu/127.19.84.62@KRBTEST.COM'} at 127.0.0.1:43000
I20260131 03:13:46.711434 7230 tablet_replica.cc:333] T 75a95ee19134477191da3b0065aafadf P 77302c2270ca45e3b6e1bef96e62ac96: stopping tablet replica
I20260131 03:13:46.711514 7230 raft_consensus.cc:2243] T 75a95ee19134477191da3b0065aafadf P 77302c2270ca45e3b6e1bef96e62ac96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:13:46.711637 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 6786
I20260131 03:13:46.712319 7230 raft_consensus.cc:2272] T 75a95ee19134477191da3b0065aafadf P 77302c2270ca45e3b6e1bef96e62ac96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:13:46.712782 7230 ts_tablet_manager.cc:1916] T 75a95ee19134477191da3b0065aafadf P 77302c2270ca45e3b6e1bef96e62ac96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:13:46.712814 7229 tablet_replica.cc:333] T 75a95ee19134477191da3b0065aafadf P ed96dc32101246e3b98f2fd4cb54dfa7: stopping tablet replica
I20260131 03:13:46.712882 7229 raft_consensus.cc:2243] T 75a95ee19134477191da3b0065aafadf P ed96dc32101246e3b98f2fd4cb54dfa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:13:46.712978 7229 raft_consensus.cc:2272] T 75a95ee19134477191da3b0065aafadf P ed96dc32101246e3b98f2fd4cb54dfa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:13:46.713425 7229 ts_tablet_manager.cc:1916] T 75a95ee19134477191da3b0065aafadf P ed96dc32101246e3b98f2fd4cb54dfa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:13:46.714596 7230 ts_tablet_manager.cc:1929] T 75a95ee19134477191da3b0065aafadf P 77302c2270ca45e3b6e1bef96e62ac96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:13:46.714669 7230 log.cc:1199] T 75a95ee19134477191da3b0065aafadf P 77302c2270ca45e3b6e1bef96e62ac96: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-1/wal/wals/75a95ee19134477191da3b0065aafadf
I20260131 03:13:46.714984 7230 ts_tablet_manager.cc:1950] T 75a95ee19134477191da3b0065aafadf P 77302c2270ca45e3b6e1bef96e62ac96: Deleting consensus metadata
I20260131 03:13:46.714998 7229 ts_tablet_manager.cc:1929] T 75a95ee19134477191da3b0065aafadf P ed96dc32101246e3b98f2fd4cb54dfa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:13:46.715059 7229 log.cc:1199] T 75a95ee19134477191da3b0065aafadf P ed96dc32101246e3b98f2fd4cb54dfa7: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/75a95ee19134477191da3b0065aafadf
I20260131 03:13:46.715396 7229 ts_tablet_manager.cc:1950] T 75a95ee19134477191da3b0065aafadf P ed96dc32101246e3b98f2fd4cb54dfa7: Deleting consensus metadata
I20260131 03:13:46.715418 6708 catalog_manager.cc:4985] TS 77302c2270ca45e3b6e1bef96e62ac96 (127.19.84.2:45217): tablet 75a95ee19134477191da3b0065aafadf (table test-table [id=56ee6cff1e7a489783f63d094ceb78e8]) successfully deleted
I20260131 03:13:46.715860 6709 catalog_manager.cc:4985] TS ed96dc32101246e3b98f2fd4cb54dfa7 (127.19.84.3:36467): tablet 75a95ee19134477191da3b0065aafadf (table test-table [id=56ee6cff1e7a489783f63d094ceb78e8]) successfully deleted
W20260131 03:13:46.717741 6706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.84.1:35485 (error 108)
I20260131 03:13:46.718305 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 6923
I20260131 03:13:46.718331 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:46.718197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:35485 (local address 127.0.0.1:43012)
0131 03:13:46.718263 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:46.718304 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.1:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260131 03:13:46.718436 6706 catalog_manager.cc:4712] TS 3fa8d015670d4fb398e6d6bdc0b45e4e (127.19.84.1:35485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 75a95ee19134477191da3b0065aafadf: Network error: Client connection negotiation failed: client connection to 127.19.84.1:35485: connect: Connection refused (error 111)
I20260131 03:13:46.723286 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 7059
I20260131 03:13:46.727898 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 6690
2026-01-31T03:13:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2935 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-01-31T03:13:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:13:46Z Disabled control of system clock
I20260131 03:13:46.756165 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:38517
--webserver_interface=127.19.84.62
--webserver_port=0
--builtin_ntp_servers=127.19.84.20:33575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:38517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260131 03:13:46.837317 7236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:46.837498 7236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:46.837520 7236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:46.838927 7236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:13:46.838968 7236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:46.838981 7236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:46.839000 7236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:13:46.839013 7236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:13:46.840678 7236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:33575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:38517
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:38517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:46.840885 7236 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:46.841078 7236 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:46.843730 7241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:46.843744 7244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:46.843777 7242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:46.843974 7236 server_base.cc:1060] running on GCE node
I20260131 03:13:46.844573 7236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:46.844846 7236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:46.846040 7236 hybrid_clock.cc:648] HybridClock initialized: now 1769829226846009 us; error 42 us; skew 500 ppm
I20260131 03:13:46.847865 7236 webserver.cc:492] Webserver started at http://127.19.84.62:37233/ using document root <none> and password file <none>
I20260131 03:13:46.848109 7236 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:46.848182 7236 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:46.848395 7236 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:46.849825 7236 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "45c3466a3d53457cb15820d095ae62d6"
format_stamp: "Formatted at 2026-01-31 03:13:46 on dist-test-slave-5nnp"
server_key: "787cd38ddb99f28c5306f7610d1a58d7"
server_key_iv: "e75db616a7e00725930f22a066d229f6"
server_key_version: "encryptionkey@0"
I20260131 03:13:46.850159 7236 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "45c3466a3d53457cb15820d095ae62d6"
format_stamp: "Formatted at 2026-01-31 03:13:46 on dist-test-slave-5nnp"
server_key: "787cd38ddb99f28c5306f7610d1a58d7"
server_key_iv: "e75db616a7e00725930f22a066d229f6"
server_key_version: "encryptionkey@0"
I20260131 03:13:46.851861 7236 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260131 03:13:46.852681 7250 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:46.852892 7236 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:13:46.852988 7236 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "45c3466a3d53457cb15820d095ae62d6"
format_stamp: "Formatted at 2026-01-31 03:13:46 on dist-test-slave-5nnp"
server_key: "787cd38ddb99f28c5306f7610d1a58d7"
server_key_iv: "e75db616a7e00725930f22a066d229f6"
server_key_version: "encryptionkey@0"
I20260131 03:13:46.853070 7236 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:46.860527 7236 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:46.860819 7236 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:46.860953 7236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:46.865358 7236 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:38517
I20260131 03:13:46.865438 7302 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:38517 every 8 connection(s)
I20260131 03:13:46.865806 7236 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:13:46.866565 7303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:46.869566 7303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45c3466a3d53457cb15820d095ae62d6: Bootstrap starting.
I20260131 03:13:46.870082 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 7236
I20260131 03:13:46.870180 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:13:46.870270 7303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45c3466a3d53457cb15820d095ae62d6: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:46.870385 19792 external_mini_cluster.cc:1468] Setting key 5256f9a7f1b3d8a6792cdd4b273072fd
I20260131 03:13:46.870572 7303 log.cc:826] T 00000000000000000000000000000000 P 45c3466a3d53457cb15820d095ae62d6: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:46.871706 7303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45c3466a3d53457cb15820d095ae62d6: No bootstrap required, opened a new log
I20260131 03:13:46.873695 7303 raft_consensus.cc:359] T 00000000000000000000000000000000 P 45c3466a3d53457cb15820d095ae62d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c3466a3d53457cb15820d095ae62d6" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 38517 } }
I20260131 03:13:46.873865 7303 raft_consensus.cc:385] T 00000000000000000000000000000000 P 45c3466a3d53457cb15820d095ae62d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:46.873898 7303 raft_consensus.cc:740] T 00000000000000000000000000000000 P 45c3466a3d53457cb15820d095ae62d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45c3466a3d53457cb15820d095ae62d6, State: Initialized, Role: FOLLOWER
I20260131 03:13:46.874033 7303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45c3466a3d53457cb15820d095ae62d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c3466a3d53457cb15820d095ae62d6" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 38517 } }
I20260131 03:13:46.874157 7303 raft_consensus.cc:399] T 00000000000000000000000000000000 P 45c3466a3d53457cb15820d095ae62d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:13:46.874205 7303 raft_consensus.cc:493] T 00000000000000000000000000000000 P 45c3466a3d53457cb15820d095ae62d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:13:46.874276 7303 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 45c3466a3d53457cb15820d095ae62d6 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:46.875087 7303 raft_consensus.cc:515] T 00000000000000000000000000000000 P 45c3466a3d53457cb15820d095ae62d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c3466a3d53457cb15820d095ae62d6" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 38517 } }
I20260131 03:13:46.875234 7303 leader_election.cc:304] T 00000000000000000000000000000000 P 45c3466a3d53457cb15820d095ae62d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45c3466a3d53457cb15820d095ae62d6; no voters:
I20260131 03:13:46.875475 7303 leader_election.cc:290] T 00000000000000000000000000000000 P 45c3466a3d53457cb15820d095ae62d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:13:46.875510 7308 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 45c3466a3d53457cb15820d095ae62d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:13:46.875619 7308 raft_consensus.cc:697] T 00000000000000000000000000000000 P 45c3466a3d53457cb15820d095ae62d6 [term 1 LEADER]: Becoming Leader. State: Replica: 45c3466a3d53457cb15820d095ae62d6, State: Running, Role: LEADER
I20260131 03:13:46.875702 7308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45c3466a3d53457cb15820d095ae62d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c3466a3d53457cb15820d095ae62d6" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 38517 } }
I20260131 03:13:46.875737 7303 sys_catalog.cc:565] T 00000000000000000000000000000000 P 45c3466a3d53457cb15820d095ae62d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:13:46.876030 7308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45c3466a3d53457cb15820d095ae62d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45c3466a3d53457cb15820d095ae62d6. Latest consensus state: current_term: 1 leader_uuid: "45c3466a3d53457cb15820d095ae62d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c3466a3d53457cb15820d095ae62d6" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 38517 } } }
I20260131 03:13:46.876096 7308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45c3466a3d53457cb15820d095ae62d6 [sys.catalog]: This master's current role is: LEADER
I20260131 03:13:46.876273 7306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:46.870992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41294 (local address 127.19.84.62:38517)
0131 03:13:46.871233 (+ 241us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:46.871241 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:46.871517 (+ 276us) server_negotiation.cc:408] Connection header received
0131 03:13:46.872139 (+ 622us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:46.872150 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:46.872261 (+ 111us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:46.872422 (+ 161us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:46.872821 (+ 399us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.873973 (+ 1152us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:46.874734 (+ 761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:46.875034 (+ 300us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:46.875476 (+ 442us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:46.875486 (+ 10us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:46.875487 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:13:46.875755 (+ 268us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:46.875795 (+ 40us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:13:46.875819 (+ 24us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:46.875920 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:46.876015 (+ 95us) server_negotiation.cc:300] Negotiation successful
0131 03:13:46.876087 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":109,"threads_started":1}
I20260131 03:13:46.876283 7310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45c3466a3d53457cb15820d095ae62d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45c3466a3d53457cb15820d095ae62d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c3466a3d53457cb15820d095ae62d6" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 38517 } } }
I20260131 03:13:46.876456 7310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45c3466a3d53457cb15820d095ae62d6 [sys.catalog]: This master's current role is: LEADER
I20260131 03:13:46.876866 7312 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:13:46.877733 7312 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:13:46.879429 7312 catalog_manager.cc:1357] Generated new cluster ID: db5a1612617047868a3519ad0e7f7a54
I20260131 03:13:46.879488 7312 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:13:46.883358 7312 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:13:46.884128 7312 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:13:46.898150 7312 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 45c3466a3d53457cb15820d095ae62d6: Generated new TSK 0
I20260131 03:13:46.898382 7312 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:13:46.901396 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.1:0
--local_ip_for_outbound_sockets=127.19.84.1
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:38517
--builtin_ntp_servers=127.19.84.20:33575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260131 03:13:46.982191 7327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:46.982416 7327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:46.982455 7327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:46.983949 7327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:46.984027 7327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:46.984069 7327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.1
I20260131 03:13:46.985734 7327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:33575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.84.1
--webserver_port=0
--tserver_master_addrs=127.19.84.62:38517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.1
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:46.986014 7327 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:46.986799 7327 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:46.990120 7333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:46.990159 7335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:46.990131 7332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:46.990487 7327 server_base.cc:1060] running on GCE node
I20260131 03:13:46.990675 7327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:46.990953 7327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:46.992163 7327 hybrid_clock.cc:648] HybridClock initialized: now 1769829226992085 us; error 97 us; skew 500 ppm
I20260131 03:13:46.993636 7327 webserver.cc:492] Webserver started at http://127.19.84.1:33263/ using document root <none> and password file <none>
I20260131 03:13:46.993878 7327 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:46.993938 7327 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:46.994052 7327 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:46.995347 7327 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-0/data/instance:
uuid: "33c978d585f149239205a73117e1434b"
format_stamp: "Formatted at 2026-01-31 03:13:46 on dist-test-slave-5nnp"
server_key: "e9be35ae89aee6f77b3d263116950ef8"
server_key_iv: "03457f5e62635b932146bada753412bf"
server_key_version: "encryptionkey@0"
I20260131 03:13:46.995749 7327 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance:
uuid: "33c978d585f149239205a73117e1434b"
format_stamp: "Formatted at 2026-01-31 03:13:46 on dist-test-slave-5nnp"
server_key: "e9be35ae89aee6f77b3d263116950ef8"
server_key_iv: "03457f5e62635b932146bada753412bf"
server_key_version: "encryptionkey@0"
I20260131 03:13:46.997345 7327 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:13:46.998356 7341 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:46.998632 7327 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:13:46.998725 7327 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-0/wal
uuid: "33c978d585f149239205a73117e1434b"
format_stamp: "Formatted at 2026-01-31 03:13:46 on dist-test-slave-5nnp"
server_key: "e9be35ae89aee6f77b3d263116950ef8"
server_key_iv: "03457f5e62635b932146bada753412bf"
server_key_version: "encryptionkey@0"
I20260131 03:13:46.998813 7327 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:47.015743 7327 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:47.016067 7327 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:47.016304 7327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:47.016634 7327 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:13:47.017047 7327 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:13:47.017303 7327 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:47.017377 7327 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:13:47.017412 7327 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:47.025076 7327 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.1:39271
I20260131 03:13:47.025180 7456 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.1:39271 every 8 connection(s)
I20260131 03:13:47.025511 7327 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-0/data/info.pb
I20260131 03:13:47.026201 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 7327
I20260131 03:13:47.026340 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-0/wal/instance
I20260131 03:13:47.026527 19792 external_mini_cluster.cc:1468] Setting key c3941f84a384ccdd51170c1b3cbf24d2
I20260131 03:13:47.027643 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.2:0
--local_ip_for_outbound_sockets=127.19.84.2
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:38517
--builtin_ntp_servers=127.19.84.20:33575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260131 03:13:47.030912 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:47.026351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:38517 (local address 127.19.84.1:55551)
0131 03:13:47.026696 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:47.026761 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:47.027412 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:47.027807 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:47.027812 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:47.027824 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:13:47.028175 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:47.028177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.028902 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:47.028903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:47.029451 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:47.029454 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.029653 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:47.030373 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:13:47.030374 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:13:47.030408 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:13:47.030431 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:13:47.030634 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:13:47.030638 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:13:47.030641 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:13:47.030709 (+ 68us) client_negotiation.cc:240] Negotiation successful
0131 03:13:47.030780 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":75,"threads_started":1}
I20260131 03:13:47.031162 7306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:47.026412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.1:55551 (local address 127.19.84.62:38517)
0131 03:13:47.026503 (+ 91us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:47.026505 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:47.026854 (+ 349us) server_negotiation.cc:408] Connection header received
0131 03:13:47.027535 (+ 681us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:47.027537 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:47.027554 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:47.027589 (+ 35us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:47.028269 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.028793 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:47.029596 (+ 803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.029784 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:47.030531 (+ 747us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:47.030535 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:47.030535 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:13:47.030540 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:47.030557 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:13:47.030561 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:47.030611 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:47.031064 (+ 453us) server_negotiation.cc:300] Negotiation successful
0131 03:13:47.031110 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260131 03:13:47.031630 7457 heartbeater.cc:344] Connected to a master server at 127.19.84.62:38517
I20260131 03:13:47.031733 7457 heartbeater.cc:461] Registering TS with master...
I20260131 03:13:47.031973 7457 heartbeater.cc:507] Master 127.19.84.62:38517 requested a full tablet report, sending...
I20260131 03:13:47.032541 7267 ts_manager.cc:194] Registered new tserver with Master: 33c978d585f149239205a73117e1434b (127.19.84.1:39271)
I20260131 03:13:47.033499 7267 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.19.84.1:55551
W20260131 03:13:47.111613 7460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:47.111764 7460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:47.111786 7460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:47.113126 7460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:47.113173 7460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:47.113195 7460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.2
I20260131 03:13:47.114657 7460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:33575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.84.2
--webserver_port=0
--tserver_master_addrs=127.19.84.62:38517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.2
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:47.114835 7460 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:47.115027 7460 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:47.117645 7468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:47.117707 7465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:47.117869 7460 server_base.cc:1060] running on GCE node
W20260131 03:13:47.117908 7466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:47.118095 7460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:47.118424 7460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:47.119630 7460 hybrid_clock.cc:648] HybridClock initialized: now 1769829227119609 us; error 32 us; skew 500 ppm
I20260131 03:13:47.120770 7460 webserver.cc:492] Webserver started at http://127.19.84.2:39689/ using document root <none> and password file <none>
I20260131 03:13:47.120980 7460 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:47.121042 7460 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:47.121199 7460 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:47.122144 7460 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-1/data/instance:
uuid: "eed310114895489ba48985a05aa5cffc"
format_stamp: "Formatted at 2026-01-31 03:13:47 on dist-test-slave-5nnp"
server_key: "ebe0738546c59620328823045a334272"
server_key_iv: "ad0ed00521b8e369f890345e573541de"
server_key_version: "encryptionkey@0"
I20260131 03:13:47.122517 7460 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance:
uuid: "eed310114895489ba48985a05aa5cffc"
format_stamp: "Formatted at 2026-01-31 03:13:47 on dist-test-slave-5nnp"
server_key: "ebe0738546c59620328823045a334272"
server_key_iv: "ad0ed00521b8e369f890345e573541de"
server_key_version: "encryptionkey@0"
I20260131 03:13:47.123960 7460 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:13:47.124688 7474 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:47.124871 7460 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260131 03:13:47.124929 7460 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-1/wal
uuid: "eed310114895489ba48985a05aa5cffc"
format_stamp: "Formatted at 2026-01-31 03:13:47 on dist-test-slave-5nnp"
server_key: "ebe0738546c59620328823045a334272"
server_key_iv: "ad0ed00521b8e369f890345e573541de"
server_key_version: "encryptionkey@0"
I20260131 03:13:47.124974 7460 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:47.135923 7460 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:47.136116 7460 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:47.136209 7460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:47.136361 7460 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:13:47.136656 7460 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:13:47.136689 7460 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:47.136744 7460 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:13:47.136772 7460 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:47.142885 7460 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.2:44927
I20260131 03:13:47.142932 7588 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.2:44927 every 8 connection(s)
I20260131 03:13:47.143214 7460 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-1/data/info.pb
I20260131 03:13:47.144661 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 7460
I20260131 03:13:47.144773 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-1/wal/instance
I20260131 03:13:47.144970 19792 external_mini_cluster.cc:1468] Setting key c1ca59af6cefbc0a18a2092e70196858
I20260131 03:13:47.145948 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.84.3:0
--local_ip_for_outbound_sockets=127.19.84.3
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:38517
--builtin_ntp_servers=127.19.84.20:33575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260131 03:13:47.147771 7306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:47.143961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.2:59153 (local address 127.19.84.62:38517)
0131 03:13:47.144043 (+ 82us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:47.144045 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:47.144220 (+ 175us) server_negotiation.cc:408] Connection header received
0131 03:13:47.144698 (+ 478us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:47.144700 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:47.144717 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:47.144798 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:47.145409 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.146057 (+ 648us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:47.146651 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.146831 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:47.147469 (+ 638us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:47.147473 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:47.147473 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:13:47.147478 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:47.147493 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:13:47.147496 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:47.147552 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:47.147650 (+ 98us) server_negotiation.cc:300] Negotiation successful
0131 03:13:47.147684 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260131 03:13:47.147902 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:47.143921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:38517 (local address 127.19.84.2:59153)
0131 03:13:47.144112 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:47.144141 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:47.144595 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:47.144938 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:47.144942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:47.144954 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:13:47.145341 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:47.145344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.146171 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:47.146172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:47.146561 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:47.146563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.146711 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:47.147318 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:13:47.147318 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:13:47.147343 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:13:47.147359 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:13:47.147555 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:13:47.147559 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:13:47.147561 (+ 2us) client_negotiation.cc:769] Sending connection context
0131 03:13:47.147655 (+ 94us) client_negotiation.cc:240] Negotiation successful
0131 03:13:47.147732 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260131 03:13:47.148357 7589 heartbeater.cc:344] Connected to a master server at 127.19.84.62:38517
I20260131 03:13:47.148451 7589 heartbeater.cc:461] Registering TS with master...
I20260131 03:13:47.148663 7589 heartbeater.cc:507] Master 127.19.84.62:38517 requested a full tablet report, sending...
I20260131 03:13:47.149044 7267 ts_manager.cc:194] Registered new tserver with Master: eed310114895489ba48985a05aa5cffc (127.19.84.2:44927)
I20260131 03:13:47.149473 7267 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.19.84.2:59153
W20260131 03:13:47.227303 7592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:47.227538 7592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:47.227561 7592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:47.229144 7592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:47.229192 7592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:47.229215 7592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.84.3
I20260131 03:13:47.230764 7592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:33575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.84.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.84.3
--webserver_port=0
--tserver_master_addrs=127.19.84.62:38517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.84.3
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:47.231006 7592 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:47.231248 7592 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:47.233836 7598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:47.233961 7592 server_base.cc:1060] running on GCE node
W20260131 03:13:47.233824 7597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:47.233824 7600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:47.234266 7592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:47.234485 7592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:47.235615 7592 hybrid_clock.cc:648] HybridClock initialized: now 1769829227235594 us; error 38 us; skew 500 ppm
I20260131 03:13:47.236712 7592 webserver.cc:492] Webserver started at http://127.19.84.3:42591/ using document root <none> and password file <none>
I20260131 03:13:47.236912 7592 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:47.236971 7592 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:47.237151 7592 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:47.238070 7592 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-2/data/instance:
uuid: "c670839c760b4c489169b96c0f8ee153"
format_stamp: "Formatted at 2026-01-31 03:13:47 on dist-test-slave-5nnp"
server_key: "a25ba4104ac2abfcc5ab765c8c4fee41"
server_key_iv: "721be20a22cc4846d890307d14554fbc"
server_key_version: "encryptionkey@0"
I20260131 03:13:47.238385 7592 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance:
uuid: "c670839c760b4c489169b96c0f8ee153"
format_stamp: "Formatted at 2026-01-31 03:13:47 on dist-test-slave-5nnp"
server_key: "a25ba4104ac2abfcc5ab765c8c4fee41"
server_key_iv: "721be20a22cc4846d890307d14554fbc"
server_key_version: "encryptionkey@0"
I20260131 03:13:47.239603 7592 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260131 03:13:47.240341 7606 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:47.240559 7592 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:13:47.240635 7592 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-2/wal
uuid: "c670839c760b4c489169b96c0f8ee153"
format_stamp: "Formatted at 2026-01-31 03:13:47 on dist-test-slave-5nnp"
server_key: "a25ba4104ac2abfcc5ab765c8c4fee41"
server_key_iv: "721be20a22cc4846d890307d14554fbc"
server_key_version: "encryptionkey@0"
I20260131 03:13:47.240708 7592 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:47.249645 7592 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:47.249861 7592 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:47.249974 7592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:47.250166 7592 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260131 03:13:47.250464 7592 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260131 03:13:47.250517 7592 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:47.250558 7592 ts_tablet_manager.cc:616] Registered 0 tablets
I20260131 03:13:47.250582 7592 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:47.256206 7592 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.3:39073
I20260131 03:13:47.256295 7719 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.3:39073 every 8 connection(s)
I20260131 03:13:47.256577 7592 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-2/data/info.pb
I20260131 03:13:47.260466 7306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:47.257288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:34655 (local address 127.19.84.62:38517)
0131 03:13:47.257368 (+ 80us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:47.257370 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:47.257532 (+ 162us) server_negotiation.cc:408] Connection header received
0131 03:13:47.257953 (+ 421us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:47.257955 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:47.257965 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:47.258040 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:47.258465 (+ 425us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.258949 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:47.259585 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.259711 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:47.260216 (+ 505us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:47.260220 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:47.260221 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:13:47.260226 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:47.260242 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:13:47.260245 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:47.260299 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:47.260385 (+ 86us) server_negotiation.cc:300] Negotiation successful
0131 03:13:47.260411 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260131 03:13:47.260587 7722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:47.257261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:38517 (local address 127.19.84.3:34655)
0131 03:13:47.257437 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:47.257460 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:47.257884 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:47.258082 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:47.258085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:47.258092 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:13:47.258371 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:47.258373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.259033 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:47.259033 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:47.259511 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:47.259513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.259612 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:47.260083 (+ 471us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0131 03:13:47.260084 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:13:47.260101 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0131 03:13:47.260116 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:13:47.260308 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:13:47.260326 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:13:47.260328 (+ 2us) client_negotiation.cc:769] Sending connection context
0131 03:13:47.260381 (+ 53us) client_negotiation.cc:240] Negotiation successful
0131 03:13:47.260437 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260131 03:13:47.260915 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 7592
I20260131 03:13:47.261040 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-2/wal/instance
I20260131 03:13:47.261205 7720 heartbeater.cc:344] Connected to a master server at 127.19.84.62:38517
I20260131 03:13:47.261303 7720 heartbeater.cc:461] Registering TS with master...
I20260131 03:13:47.261389 19792 external_mini_cluster.cc:1468] Setting key 88718e3a60e881d6ef815c76a665c46b
I20260131 03:13:47.261518 7720 heartbeater.cc:507] Master 127.19.84.62:38517 requested a full tablet report, sending...
I20260131 03:13:47.262007 7266 ts_manager.cc:194] Registered new tserver with Master: c670839c760b4c489169b96c0f8ee153 (127.19.84.3:39073)
I20260131 03:13:47.262406 19792 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260131 03:13:47.262558 7266 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.19.84.3:34655
I20260131 03:13:47.266104 7306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:47.263840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41310 (local address 127.19.84.62:38517)
0131 03:13:47.263894 (+ 54us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:47.263895 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:47.264002 (+ 107us) server_negotiation.cc:408] Connection header received
0131 03:13:47.264083 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:47.264084 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:47.264092 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:47.264139 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:47.264426 (+ 287us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.264846 (+ 420us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:47.265384 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.265493 (+ 109us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:47.265903 (+ 410us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:47.265905 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:47.265906 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:13:47.265909 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:47.265918 (+ 9us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:13:47.265921 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:47.265966 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:47.266035 (+ 69us) server_negotiation.cc:300] Negotiation successful
0131 03:13:47.266056 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260131 03:13:47.267709 7266 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260131 03:13:47.268859 7266 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260131 03:13:47.274224 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:47.270804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.3:39073 (local address 127.0.0.1:43448)
0131 03:13:47.271356 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:47.271367 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:47.271577 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:47.271883 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:47.271884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:47.271886 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:47.272038 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:47.272040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.273137 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:47.273138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:47.273965 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:47.273967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.274039 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:47.274074 (+ 35us) client_negotiation.cc:769] Sending connection context
0131 03:13:47.274117 (+ 43us) client_negotiation.cc:240] Negotiation successful
0131 03:13:47.274154 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":120,"threads_started":1}
I20260131 03:13:47.274410 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:47.270730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:39271 (local address 127.0.0.1:49814)
0131 03:13:47.271027 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:47.271054 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:47.271112 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:47.271351 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:47.271354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:47.271356 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:47.271607 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:47.271609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.273331 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:47.273332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:47.274214 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:47.274216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.274277 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:47.274280 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:13:47.274312 (+ 32us) client_negotiation.cc:240] Negotiation successful
0131 03:13:47.274351 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":49,"threads_started":1}
I20260131 03:13:47.274537 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:47.270905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:44927 (local address 127.0.0.1:38166)
0131 03:13:47.271425 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:47.271433 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:47.271474 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:47.271692 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:47.271693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:47.271695 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:47.271887 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:47.271888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.273483 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:47.273484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:47.274267 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:47.274269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.274339 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:47.274344 (+ 5us) client_negotiation.cc:769] Sending connection context
0131 03:13:47.274379 (+ 35us) client_negotiation.cc:240] Negotiation successful
0131 03:13:47.274414 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":67,"threads_started":1}
I20260131 03:13:47.275147 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:47.271000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49814 (local address 127.19.84.1:39271)
0131 03:13:47.271172 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:47.271179 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:47.271191 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:13:47.271230 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:47.271233 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:47.271250 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:47.271334 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:47.271694 (+ 360us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.273251 (+ 1557us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:47.274537 (+ 1286us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.274976 (+ 439us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:47.275044 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:47.275069 (+ 25us) server_negotiation.cc:300] Negotiation successful
0131 03:13:47.275092 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
I20260131 03:13:47.275267 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:47.271178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.19.84.2:44927)
0131 03:13:47.271446 (+ 268us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:47.271454 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:47.271481 (+ 27us) server_negotiation.cc:408] Connection header received
0131 03:13:47.271537 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:47.271539 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:47.271554 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:47.271611 (+ 57us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:47.271979 (+ 368us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.273395 (+ 1416us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:47.274449 (+ 1054us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.275036 (+ 587us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:47.275122 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:47.275152 (+ 30us) server_negotiation.cc:300] Negotiation successful
0131 03:13:47.275191 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":103,"threads_started":1}
I20260131 03:13:47.275352 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:47.270873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43448 (local address 127.19.84.3:39073)
0131 03:13:47.271686 (+ 813us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:47.271694 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:47.271703 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:13:47.271735 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:47.271737 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:47.271754 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:47.271813 (+ 59us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:47.272118 (+ 305us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.273049 (+ 931us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:47.274051 (+ 1002us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.275144 (+ 1093us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:47.275201 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:47.275226 (+ 25us) server_negotiation.cc:300] Negotiation successful
0131 03:13:47.275262 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":88,"threads_started":1}
I20260131 03:13:47.275872 7389 tablet_service.cc:1505] Processing CreateTablet for tablet a1e27b58db9d4435bd8be9fcc8d77f2e (DEFAULT_TABLE table=test-table [id=0998907177de435482a8adc346bf2aa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:13:47.275935 7654 tablet_service.cc:1505] Processing CreateTablet for tablet a1e27b58db9d4435bd8be9fcc8d77f2e (DEFAULT_TABLE table=test-table [id=0998907177de435482a8adc346bf2aa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:13:47.275959 7523 tablet_service.cc:1505] Processing CreateTablet for tablet a1e27b58db9d4435bd8be9fcc8d77f2e (DEFAULT_TABLE table=test-table [id=0998907177de435482a8adc346bf2aa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260131 03:13:47.276189 7389 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1e27b58db9d4435bd8be9fcc8d77f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:47.276189 7654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1e27b58db9d4435bd8be9fcc8d77f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:47.276270 7523 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1e27b58db9d4435bd8be9fcc8d77f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:47.278440 7734 tablet_bootstrap.cc:492] T a1e27b58db9d4435bd8be9fcc8d77f2e P eed310114895489ba48985a05aa5cffc: Bootstrap starting.
I20260131 03:13:47.278914 7735 tablet_bootstrap.cc:492] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153: Bootstrap starting.
I20260131 03:13:47.279362 7734 tablet_bootstrap.cc:654] T a1e27b58db9d4435bd8be9fcc8d77f2e P eed310114895489ba48985a05aa5cffc: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:47.279553 7736 tablet_bootstrap.cc:492] T a1e27b58db9d4435bd8be9fcc8d77f2e P 33c978d585f149239205a73117e1434b: Bootstrap starting.
I20260131 03:13:47.279578 7735 tablet_bootstrap.cc:654] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:47.279707 7734 log.cc:826] T a1e27b58db9d4435bd8be9fcc8d77f2e P eed310114895489ba48985a05aa5cffc: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:47.279846 7735 log.cc:826] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:47.280200 7736 tablet_bootstrap.cc:654] T a1e27b58db9d4435bd8be9fcc8d77f2e P 33c978d585f149239205a73117e1434b: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:47.280478 7736 log.cc:826] T a1e27b58db9d4435bd8be9fcc8d77f2e P 33c978d585f149239205a73117e1434b: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:47.280534 7734 tablet_bootstrap.cc:492] T a1e27b58db9d4435bd8be9fcc8d77f2e P eed310114895489ba48985a05aa5cffc: No bootstrap required, opened a new log
I20260131 03:13:47.280606 7734 ts_tablet_manager.cc:1403] T a1e27b58db9d4435bd8be9fcc8d77f2e P eed310114895489ba48985a05aa5cffc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260131 03:13:47.280617 7735 tablet_bootstrap.cc:492] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153: No bootstrap required, opened a new log
I20260131 03:13:47.280718 7735 ts_tablet_manager.cc:1403] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:13:47.281178 7736 tablet_bootstrap.cc:492] T a1e27b58db9d4435bd8be9fcc8d77f2e P 33c978d585f149239205a73117e1434b: No bootstrap required, opened a new log
I20260131 03:13:47.281258 7736 ts_tablet_manager.cc:1403] T a1e27b58db9d4435bd8be9fcc8d77f2e P 33c978d585f149239205a73117e1434b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260131 03:13:47.282177 7734 raft_consensus.cc:359] T a1e27b58db9d4435bd8be9fcc8d77f2e P eed310114895489ba48985a05aa5cffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c978d585f149239205a73117e1434b" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39271 } } peers { permanent_uuid: "eed310114895489ba48985a05aa5cffc" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 44927 } } peers { permanent_uuid: "c670839c760b4c489169b96c0f8ee153" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39073 } }
I20260131 03:13:47.282189 7735 raft_consensus.cc:359] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c978d585f149239205a73117e1434b" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39271 } } peers { permanent_uuid: "eed310114895489ba48985a05aa5cffc" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 44927 } } peers { permanent_uuid: "c670839c760b4c489169b96c0f8ee153" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39073 } }
I20260131 03:13:47.282322 7735 raft_consensus.cc:385] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:47.282322 7734 raft_consensus.cc:385] T a1e27b58db9d4435bd8be9fcc8d77f2e P eed310114895489ba48985a05aa5cffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:47.282353 7735 raft_consensus.cc:740] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c670839c760b4c489169b96c0f8ee153, State: Initialized, Role: FOLLOWER
I20260131 03:13:47.282353 7734 raft_consensus.cc:740] T a1e27b58db9d4435bd8be9fcc8d77f2e P eed310114895489ba48985a05aa5cffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed310114895489ba48985a05aa5cffc, State: Initialized, Role: FOLLOWER
I20260131 03:13:47.282440 7734 consensus_queue.cc:260] T a1e27b58db9d4435bd8be9fcc8d77f2e P eed310114895489ba48985a05aa5cffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c978d585f149239205a73117e1434b" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39271 } } peers { permanent_uuid: "eed310114895489ba48985a05aa5cffc" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 44927 } } peers { permanent_uuid: "c670839c760b4c489169b96c0f8ee153" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39073 } }
I20260131 03:13:47.282440 7735 consensus_queue.cc:260] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c978d585f149239205a73117e1434b" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39271 } } peers { permanent_uuid: "eed310114895489ba48985a05aa5cffc" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 44927 } } peers { permanent_uuid: "c670839c760b4c489169b96c0f8ee153" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39073 } }
I20260131 03:13:47.282671 7735 ts_tablet_manager.cc:1434] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:13:47.282671 7734 ts_tablet_manager.cc:1434] T a1e27b58db9d4435bd8be9fcc8d77f2e P eed310114895489ba48985a05aa5cffc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260131 03:13:47.282630 7736 raft_consensus.cc:359] T a1e27b58db9d4435bd8be9fcc8d77f2e P 33c978d585f149239205a73117e1434b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c978d585f149239205a73117e1434b" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39271 } } peers { permanent_uuid: "eed310114895489ba48985a05aa5cffc" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 44927 } } peers { permanent_uuid: "c670839c760b4c489169b96c0f8ee153" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39073 } }
I20260131 03:13:47.282791 7720 heartbeater.cc:499] Master 127.19.84.62:38517 was elected leader, sending a full tablet report...
I20260131 03:13:47.282840 7589 heartbeater.cc:499] Master 127.19.84.62:38517 was elected leader, sending a full tablet report...
I20260131 03:13:47.283037 7736 raft_consensus.cc:385] T a1e27b58db9d4435bd8be9fcc8d77f2e P 33c978d585f149239205a73117e1434b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:47.283102 7736 raft_consensus.cc:740] T a1e27b58db9d4435bd8be9fcc8d77f2e P 33c978d585f149239205a73117e1434b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c978d585f149239205a73117e1434b, State: Initialized, Role: FOLLOWER
I20260131 03:13:47.283190 7736 consensus_queue.cc:260] T a1e27b58db9d4435bd8be9fcc8d77f2e P 33c978d585f149239205a73117e1434b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c978d585f149239205a73117e1434b" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39271 } } peers { permanent_uuid: "eed310114895489ba48985a05aa5cffc" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 44927 } } peers { permanent_uuid: "c670839c760b4c489169b96c0f8ee153" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39073 } }
I20260131 03:13:47.283434 7736 ts_tablet_manager.cc:1434] T a1e27b58db9d4435bd8be9fcc8d77f2e P 33c978d585f149239205a73117e1434b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260131 03:13:47.283582 7457 heartbeater.cc:499] Master 127.19.84.62:38517 was elected leader, sending a full tablet report...
I20260131 03:13:47.308286 7740 raft_consensus.cc:493] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260131 03:13:47.308413 7740 raft_consensus.cc:515] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c978d585f149239205a73117e1434b" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39271 } } peers { permanent_uuid: "eed310114895489ba48985a05aa5cffc" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 44927 } } peers { permanent_uuid: "c670839c760b4c489169b96c0f8ee153" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39073 } }
I20260131 03:13:47.308717 7740 leader_election.cc:290] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33c978d585f149239205a73117e1434b (127.19.84.1:39271), eed310114895489ba48985a05aa5cffc (127.19.84.2:44927)
I20260131 03:13:47.311002 7722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:47.308870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:39271 (local address 127.19.84.3:43007)
0131 03:13:47.308939 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:47.308958 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:47.309025 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:47.309189 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:47.309191 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:47.309193 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:47.309432 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:47.309434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.310191 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:47.310192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:47.310827 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:47.310829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.310885 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:47.310888 (+ 3us) client_negotiation.cc:769] Sending connection context
0131 03:13:47.310911 (+ 23us) client_negotiation.cc:240] Negotiation successful
0131 03:13:47.310934 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260131 03:13:47.311530 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:47.308979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:43007 (local address 127.19.84.1:39271)
0131 03:13:47.309030 (+ 51us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:47.309032 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:47.309066 (+ 34us) server_negotiation.cc:408] Connection header received
0131 03:13:47.309096 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:47.309097 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:47.309111 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:47.309163 (+ 52us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:47.309528 (+ 365us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.310115 (+ 587us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:47.310922 (+ 807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.311401 (+ 479us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:47.311409 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:47.311432 (+ 23us) server_negotiation.cc:300] Negotiation successful
0131 03:13:47.311462 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260131 03:13:47.311760 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:47.309009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.2:44927 (local address 127.19.84.3:54227)
0131 03:13:47.309213 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:47.309223 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:13:47.309281 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:13:47.309424 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:13:47.309424 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:13:47.309426 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0131 03:13:47.309699 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:47.309701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.310619 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:13:47.310620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:13:47.311513 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:13:47.311515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.311599 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:47.311603 (+ 4us) client_negotiation.cc:769] Sending connection context
0131 03:13:47.311639 (+ 36us) client_negotiation.cc:240] Negotiation successful
0131 03:13:47.311681 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":127,"threads_started":1}
I20260131 03:13:47.311820 7410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1e27b58db9d4435bd8be9fcc8d77f2e" candidate_uuid: "c670839c760b4c489169b96c0f8ee153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33c978d585f149239205a73117e1434b" is_pre_election: true
I20260131 03:13:47.311985 7410 raft_consensus.cc:2468] T a1e27b58db9d4435bd8be9fcc8d77f2e P 33c978d585f149239205a73117e1434b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c670839c760b4c489169b96c0f8ee153 in term 0.
I20260131 03:13:47.312219 7610 leader_election.cc:304] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33c978d585f149239205a73117e1434b, c670839c760b4c489169b96c0f8ee153; no voters:
I20260131 03:13:47.312250 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:47.309098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.84.3:54227 (local address 127.19.84.2:44927)
0131 03:13:47.309168 (+ 70us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:47.309170 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:47.309298 (+ 128us) server_negotiation.cc:408] Connection header received
0131 03:13:47.309339 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:47.309340 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:47.309352 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:47.309412 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0131 03:13:47.309777 (+ 365us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.310352 (+ 575us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:47.311626 (+ 1274us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.312140 (+ 514us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:47.312148 (+ 8us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:47.312169 (+ 21us) server_negotiation.cc:300] Negotiation successful
0131 03:13:47.312205 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260131 03:13:47.312342 7740 raft_consensus.cc:2804] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260131 03:13:47.312425 7740 raft_consensus.cc:493] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260131 03:13:47.312465 7740 raft_consensus.cc:3060] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:47.312493 7543 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1e27b58db9d4435bd8be9fcc8d77f2e" candidate_uuid: "c670839c760b4c489169b96c0f8ee153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed310114895489ba48985a05aa5cffc" is_pre_election: true
I20260131 03:13:47.312647 7543 raft_consensus.cc:2468] T a1e27b58db9d4435bd8be9fcc8d77f2e P eed310114895489ba48985a05aa5cffc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c670839c760b4c489169b96c0f8ee153 in term 0.
I20260131 03:13:47.313289 7740 raft_consensus.cc:515] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c978d585f149239205a73117e1434b" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39271 } } peers { permanent_uuid: "eed310114895489ba48985a05aa5cffc" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 44927 } } peers { permanent_uuid: "c670839c760b4c489169b96c0f8ee153" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39073 } }
I20260131 03:13:47.313438 7740 leader_election.cc:290] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153 [CANDIDATE]: Term 1 election: Requested vote from peers 33c978d585f149239205a73117e1434b (127.19.84.1:39271), eed310114895489ba48985a05aa5cffc (127.19.84.2:44927)
I20260131 03:13:47.313632 7410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1e27b58db9d4435bd8be9fcc8d77f2e" candidate_uuid: "c670839c760b4c489169b96c0f8ee153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33c978d585f149239205a73117e1434b"
I20260131 03:13:47.313695 7543 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1e27b58db9d4435bd8be9fcc8d77f2e" candidate_uuid: "c670839c760b4c489169b96c0f8ee153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed310114895489ba48985a05aa5cffc"
I20260131 03:13:47.313730 7410 raft_consensus.cc:3060] T a1e27b58db9d4435bd8be9fcc8d77f2e P 33c978d585f149239205a73117e1434b [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:47.313771 7543 raft_consensus.cc:3060] T a1e27b58db9d4435bd8be9fcc8d77f2e P eed310114895489ba48985a05aa5cffc [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:47.314666 7410 raft_consensus.cc:2468] T a1e27b58db9d4435bd8be9fcc8d77f2e P 33c978d585f149239205a73117e1434b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c670839c760b4c489169b96c0f8ee153 in term 1.
I20260131 03:13:47.314742 7543 raft_consensus.cc:2468] T a1e27b58db9d4435bd8be9fcc8d77f2e P eed310114895489ba48985a05aa5cffc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c670839c760b4c489169b96c0f8ee153 in term 1.
I20260131 03:13:47.314977 7610 leader_election.cc:304] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c670839c760b4c489169b96c0f8ee153, eed310114895489ba48985a05aa5cffc; no voters:
I20260131 03:13:47.315158 7740 raft_consensus.cc:2804] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:13:47.315335 7740 raft_consensus.cc:697] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153 [term 1 LEADER]: Becoming Leader. State: Replica: c670839c760b4c489169b96c0f8ee153, State: Running, Role: LEADER
I20260131 03:13:47.315476 7740 consensus_queue.cc:237] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c978d585f149239205a73117e1434b" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39271 } } peers { permanent_uuid: "eed310114895489ba48985a05aa5cffc" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 44927 } } peers { permanent_uuid: "c670839c760b4c489169b96c0f8ee153" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39073 } }
I20260131 03:13:47.316259 7267 catalog_manager.cc:5654] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153 reported cstate change: term changed from 0 to 1, leader changed from <none> to c670839c760b4c489169b96c0f8ee153 (127.19.84.3). New cstate: current_term: 1 leader_uuid: "c670839c760b4c489169b96c0f8ee153" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c978d585f149239205a73117e1434b" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eed310114895489ba48985a05aa5cffc" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 44927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c670839c760b4c489169b96c0f8ee153" member_type: VOTER last_known_addr { host: "127.19.84.3" port: 39073 } health_report { overall_health: HEALTHY } } }
I20260131 03:13:47.332568 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:47.330375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43460 (local address 127.19.84.3:39073)
0131 03:13:47.330431 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:47.330433 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:47.330449 (+ 16us) server_negotiation.cc:408] Connection header received
0131 03:13:47.330491 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:47.330492 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:47.330511 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:47.330571 (+ 60us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:13:47.330858 (+ 287us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.331427 (+ 569us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:47.332018 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:47.332139 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:47.332173 (+ 34us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:13:47.332260 (+ 87us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:13:47.332335 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:47.332476 (+ 141us) server_negotiation.cc:300] Negotiation successful
0131 03:13:47.332507 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260131 03:13:47.334404 7410 raft_consensus.cc:1275] T a1e27b58db9d4435bd8be9fcc8d77f2e P 33c978d585f149239205a73117e1434b [term 1 FOLLOWER]: Refusing update from remote peer c670839c760b4c489169b96c0f8ee153: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:13:47.334404 7543 raft_consensus.cc:1275] T a1e27b58db9d4435bd8be9fcc8d77f2e P eed310114895489ba48985a05aa5cffc [term 1 FOLLOWER]: Refusing update from remote peer c670839c760b4c489169b96c0f8ee153: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:13:47.334748 7744 consensus_queue.cc:1048] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eed310114895489ba48985a05aa5cffc" member_type: VOTER last_known_addr { host: "127.19.84.2" port: 44927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:13:47.334875 7740 consensus_queue.cc:1048] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33c978d585f149239205a73117e1434b" member_type: VOTER last_known_addr { host: "127.19.84.1" port: 39271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:13:47.338109 7749 mvcc.cc:204] Tried to move back new op lower bound from 7249220515158687744 to 7249220515084984320. Current Snapshot: MvccSnapshot[applied={T|T < 7249220515158687744 or (T in {7249220515158687744})}]
I20260131 03:13:47.339340 7265 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:13:47.339473 7265 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260131 03:13:47.340301 7265 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 45c3466a3d53457cb15820d095ae62d6: Sending DeleteTablet for 3 replicas of tablet a1e27b58db9d4435bd8be9fcc8d77f2e
I20260131 03:13:47.340588 7389 tablet_service.cc:1552] Processing DeleteTablet for tablet a1e27b58db9d4435bd8be9fcc8d77f2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:13:47 UTC) from {username='slave'} at 127.0.0.1:49814
I20260131 03:13:47.340644 7523 tablet_service.cc:1552] Processing DeleteTablet for tablet a1e27b58db9d4435bd8be9fcc8d77f2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:13:47 UTC) from {username='slave'} at 127.0.0.1:38166
I20260131 03:13:47.340714 7654 tablet_service.cc:1552] Processing DeleteTablet for tablet a1e27b58db9d4435bd8be9fcc8d77f2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-31 03:13:47 UTC) from {username='slave'} at 127.0.0.1:43448
I20260131 03:13:47.340806 7757 tablet_replica.cc:333] T a1e27b58db9d4435bd8be9fcc8d77f2e P 33c978d585f149239205a73117e1434b: stopping tablet replica
I20260131 03:13:47.340883 7757 raft_consensus.cc:2243] T a1e27b58db9d4435bd8be9fcc8d77f2e P 33c978d585f149239205a73117e1434b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:13:47.341053 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 7327
I20260131 03:13:47.342609 7758 tablet_replica.cc:333] T a1e27b58db9d4435bd8be9fcc8d77f2e P eed310114895489ba48985a05aa5cffc: stopping tablet replica
I20260131 03:13:47.342715 7758 raft_consensus.cc:2243] T a1e27b58db9d4435bd8be9fcc8d77f2e P eed310114895489ba48985a05aa5cffc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260131 03:13:47.342821 7758 raft_consensus.cc:2272] T a1e27b58db9d4435bd8be9fcc8d77f2e P eed310114895489ba48985a05aa5cffc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:13:47.343302 7758 ts_tablet_manager.cc:1916] T a1e27b58db9d4435bd8be9fcc8d77f2e P eed310114895489ba48985a05aa5cffc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:13:47.345356 7758 ts_tablet_manager.cc:1929] T a1e27b58db9d4435bd8be9fcc8d77f2e P eed310114895489ba48985a05aa5cffc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:13:47.345436 7758 log.cc:1199] T a1e27b58db9d4435bd8be9fcc8d77f2e P eed310114895489ba48985a05aa5cffc: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-1/wal/wals/a1e27b58db9d4435bd8be9fcc8d77f2e
I20260131 03:13:47.345700 7758 ts_tablet_manager.cc:1950] T a1e27b58db9d4435bd8be9fcc8d77f2e P eed310114895489ba48985a05aa5cffc: Deleting consensus metadata
I20260131 03:13:47.346284 7254 catalog_manager.cc:4985] TS eed310114895489ba48985a05aa5cffc (127.19.84.2:44927): tablet a1e27b58db9d4435bd8be9fcc8d77f2e (table test-table [id=0998907177de435482a8adc346bf2aa5]) successfully deleted
W20260131 03:13:47.347728 7254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.84.1:39271 (error 108)
I20260131 03:13:47.348311 7759 tablet_replica.cc:333] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153: stopping tablet replica
I20260131 03:13:47.348312 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:47.348169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.1:39271 (local address 127.0.0.1:49822)
0131 03:13:47.348227 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0131 03:13:47.348275 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.1:39271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260131 03:13:47.348428 7759 raft_consensus.cc:2243] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153 [term 1 LEADER]: Raft consensus shutting down.
W20260131 03:13:47.348539 7254 catalog_manager.cc:4712] TS 33c978d585f149239205a73117e1434b (127.19.84.1:39271): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a1e27b58db9d4435bd8be9fcc8d77f2e: Network error: Client connection negotiation failed: client connection to 127.19.84.1:39271: connect: Connection refused (error 111)
I20260131 03:13:47.348570 7759 raft_consensus.cc:2272] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260131 03:13:47.348753 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 7460
I20260131 03:13:47.349004 7759 ts_tablet_manager.cc:1916] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260131 03:13:47.350684 7759 ts_tablet_manager.cc:1929] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260131 03:13:47.350736 7759 log.cc:1199] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153: Deleting WAL directory at /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1769828849314684-19792-0/minicluster-data/ts-2/wal/wals/a1e27b58db9d4435bd8be9fcc8d77f2e
I20260131 03:13:47.350956 7759 ts_tablet_manager.cc:1950] T a1e27b58db9d4435bd8be9fcc8d77f2e P c670839c760b4c489169b96c0f8ee153: Deleting consensus metadata
I20260131 03:13:47.351413 7252 catalog_manager.cc:4985] TS c670839c760b4c489169b96c0f8ee153 (127.19.84.3:39073): tablet a1e27b58db9d4435bd8be9fcc8d77f2e (table test-table [id=0998907177de435482a8adc346bf2aa5]) successfully deleted
I20260131 03:13:47.357002 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 7592
I20260131 03:13:47.362042 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 7236
2026-01-31T03:13:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (633 ms)
[----------] 6 tests from EncryptionPolicyTest (11412 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:13:47 dist-test-slave-5nnp krb5kdc[7763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:13:47 dist-test-slave-5nnp krb5kdc[7763](info): set up 2 sockets
Jan 31 03:13:47 dist-test-slave-5nnp krb5kdc[7763](info): commencing operation
krb5kdc: starting...
W20260131 03:13:49.356004 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.973s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:13:49 dist-test-slave-5nnp krb5kdc[7763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:13:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:13:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:13:49.494969 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33813
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:43479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:13:49.571856 7779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:49.572018 7779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:49.572041 7779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:49.573465 7779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:13:49.573508 7779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:49.573522 7779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:49.573567 7779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:13:49.573597 7779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:13:49.575165 7779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33813
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:49.575372 7779 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:49.575606 7779 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:49.578135 7784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:49.578148 7785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:49.578219 7787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:49.578353 7779 server_base.cc:1060] running on GCE node
I20260131 03:13:49.578610 7779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:49.578830 7779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:49.579958 7779 hybrid_clock.cc:648] HybridClock initialized: now 1769829229579945 us; error 29 us; skew 500 ppm
Jan 31 03:13:49 dist-test-slave-5nnp krb5kdc[7763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:13:49.582067 7779 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260131 03:13:49.582630 7779 webserver.cc:492] Webserver started at http://127.0.0.1:45391/ using document root <none> and password file <none>
I20260131 03:13:49.582815 7779 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:49.582867 7779 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:49.582991 7779 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:49.583985 7779 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "f75e6fc3f8044db3b0c30edf677a3ae8"
format_stamp: "Formatted at 2026-01-31 03:13:49 on dist-test-slave-5nnp"
server_key: "4ae60289626f09f421cd54febabd1dc2"
server_key_iv: "2ccf9daa64170dd3a17e9917cff65f7b"
server_key_version: "encryptionkey@0"
I20260131 03:13:49.584308 7779 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "f75e6fc3f8044db3b0c30edf677a3ae8"
format_stamp: "Formatted at 2026-01-31 03:13:49 on dist-test-slave-5nnp"
server_key: "4ae60289626f09f421cd54febabd1dc2"
server_key_iv: "2ccf9daa64170dd3a17e9917cff65f7b"
server_key_version: "encryptionkey@0"
I20260131 03:13:49.585520 7779 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:13:49.586231 7794 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:49.586431 7779 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260131 03:13:49.586520 7779 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "f75e6fc3f8044db3b0c30edf677a3ae8"
format_stamp: "Formatted at 2026-01-31 03:13:49 on dist-test-slave-5nnp"
server_key: "4ae60289626f09f421cd54febabd1dc2"
server_key_iv: "2ccf9daa64170dd3a17e9917cff65f7b"
server_key_version: "encryptionkey@0"
I20260131 03:13:49.586594 7779 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:49.601442 7779 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:49.606108 7779 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:49.606287 7779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:49.610589 7779 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33813
I20260131 03:13:49.610618 7846 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:33813 every 8 connection(s)
I20260131 03:13:49.610935 7779 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:13:49.611593 7847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:49.614039 7847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75e6fc3f8044db3b0c30edf677a3ae8: Bootstrap starting.
I20260131 03:13:49.614624 7847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f75e6fc3f8044db3b0c30edf677a3ae8: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:49.614885 7847 log.cc:826] T 00000000000000000000000000000000 P f75e6fc3f8044db3b0c30edf677a3ae8: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:49.615635 7847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75e6fc3f8044db3b0c30edf677a3ae8: No bootstrap required, opened a new log
I20260131 03:13:49.616788 7847 raft_consensus.cc:359] T 00000000000000000000000000000000 P f75e6fc3f8044db3b0c30edf677a3ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e6fc3f8044db3b0c30edf677a3ae8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33813 } }
I20260131 03:13:49.616902 7847 raft_consensus.cc:385] T 00000000000000000000000000000000 P f75e6fc3f8044db3b0c30edf677a3ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:49.616922 7847 raft_consensus.cc:740] T 00000000000000000000000000000000 P f75e6fc3f8044db3b0c30edf677a3ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f75e6fc3f8044db3b0c30edf677a3ae8, State: Initialized, Role: FOLLOWER
I20260131 03:13:49.617076 7847 consensus_queue.cc:260] T 00000000000000000000000000000000 P f75e6fc3f8044db3b0c30edf677a3ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e6fc3f8044db3b0c30edf677a3ae8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33813 } }
I20260131 03:13:49.617127 7847 raft_consensus.cc:399] T 00000000000000000000000000000000 P f75e6fc3f8044db3b0c30edf677a3ae8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:13:49.617194 7847 raft_consensus.cc:493] T 00000000000000000000000000000000 P f75e6fc3f8044db3b0c30edf677a3ae8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:13:49.617236 7847 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f75e6fc3f8044db3b0c30edf677a3ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:49.617769 7847 raft_consensus.cc:515] T 00000000000000000000000000000000 P f75e6fc3f8044db3b0c30edf677a3ae8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e6fc3f8044db3b0c30edf677a3ae8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33813 } }
I20260131 03:13:49.617885 7847 leader_election.cc:304] T 00000000000000000000000000000000 P f75e6fc3f8044db3b0c30edf677a3ae8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f75e6fc3f8044db3b0c30edf677a3ae8; no voters:
I20260131 03:13:49.618041 7847 leader_election.cc:290] T 00000000000000000000000000000000 P f75e6fc3f8044db3b0c30edf677a3ae8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:13:49.618113 7850 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f75e6fc3f8044db3b0c30edf677a3ae8 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:13:49.618258 7847 sys_catalog.cc:565] T 00000000000000000000000000000000 P f75e6fc3f8044db3b0c30edf677a3ae8 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:13:49.618265 7850 raft_consensus.cc:697] T 00000000000000000000000000000000 P f75e6fc3f8044db3b0c30edf677a3ae8 [term 1 LEADER]: Becoming Leader. State: Replica: f75e6fc3f8044db3b0c30edf677a3ae8, State: Running, Role: LEADER
I20260131 03:13:49.618383 7850 consensus_queue.cc:237] T 00000000000000000000000000000000 P f75e6fc3f8044db3b0c30edf677a3ae8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e6fc3f8044db3b0c30edf677a3ae8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33813 } }
I20260131 03:13:49.618778 7852 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75e6fc3f8044db3b0c30edf677a3ae8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f75e6fc3f8044db3b0c30edf677a3ae8. Latest consensus state: current_term: 1 leader_uuid: "f75e6fc3f8044db3b0c30edf677a3ae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e6fc3f8044db3b0c30edf677a3ae8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33813 } } }
I20260131 03:13:49.618870 7852 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75e6fc3f8044db3b0c30edf677a3ae8 [sys.catalog]: This master's current role is: LEADER
I20260131 03:13:49.619057 7858 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:13:49.619443 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 7779
I20260131 03:13:49.619594 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:13:49.619812 19792 external_mini_cluster.cc:1468] Setting key 60cc28a3484523de0be77ed4909737e8
I20260131 03:13:49.619019 7851 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75e6fc3f8044db3b0c30edf677a3ae8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f75e6fc3f8044db3b0c30edf677a3ae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e6fc3f8044db3b0c30edf677a3ae8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33813 } } }
I20260131 03:13:49.619984 7851 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75e6fc3f8044db3b0c30edf677a3ae8 [sys.catalog]: This master's current role is: LEADER
I20260131 03:13:49.620071 7858 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:13:49.621469 7858 catalog_manager.cc:1357] Generated new cluster ID: af9b96b308d44307958b91259daf8ce8
I20260131 03:13:49.621521 7858 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Jan 31 03:13:49 dist-test-slave-5nnp krb5kdc[7763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260131 03:13:49.629012 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:49.620411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52743 (local address 127.0.0.1:33813)
0131 03:13:49.620723 (+ 312us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:49.620740 (+ 17us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:49.620764 (+ 24us) server_negotiation.cc:408] Connection header received
0131 03:13:49.620862 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:49.620867 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:49.620912 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:49.621042 (+ 130us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:49.621929 (+ 887us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:49.622694 (+ 765us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:49.623266 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:49.623548 (+ 282us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:49.625740 (+ 2192us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:49.625747 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:49.625748 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:49.625768 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:49.627516 (+ 1748us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:49.627899 (+ 383us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:49.627902 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:49.627903 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:49.627973 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:49.628183 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:49.628184 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:49.628185 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:49.628371 (+ 186us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:49.628540 (+ 169us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:49.628679 (+ 139us) server_negotiation.cc:300] Negotiation successful
0131 03:13:49.628887 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":77,"threads_started":1}
I20260131 03:13:49.629961 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:13:49.630800 7858 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:13:49.631485 7858 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:13:49.638928 7858 catalog_manager.cc:6027] T 00000000000000000000000000000000 P f75e6fc3f8044db3b0c30edf677a3ae8: Generated new TSK 0
I20260131 03:13:49.639086 7858 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:13:49.641131 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:49.631738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47047 (local address 127.0.0.1:33813)
0131 03:13:49.631774 (+ 36us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:49.631776 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:49.635537 (+ 3761us) server_negotiation.cc:408] Connection header received
0131 03:13:49.635621 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:49.635622 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:49.635633 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:49.635704 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:49.636351 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:49.636834 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:49.637378 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:49.637588 (+ 210us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:49.638655 (+ 1067us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:49.638657 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:49.638657 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:49.638663 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:49.640082 (+ 1419us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:49.640438 (+ 356us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:49.640439 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:49.640440 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:49.640468 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:49.640636 (+ 168us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:49.640637 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:49.640638 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:49.640719 (+ 81us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:49.640830 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:49.640962 (+ 132us) server_negotiation.cc:300] Negotiation successful
0131 03:13:49.641025 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260131 03:13:49.642866 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 7779
W20260131 03:13:49.647202 7761 connection.cc:537] client connection to 127.0.0.1:33813 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33813): Connection reset by peer (error 104)
2026-01-31T03:13:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2278 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:13:49 dist-test-slave-5nnp krb5kdc[7878](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:13:49 dist-test-slave-5nnp krb5kdc[7878](info): set up 2 sockets
Jan 31 03:13:49 dist-test-slave-5nnp krb5kdc[7878](info): commencing operation
krb5kdc: starting...
W20260131 03:13:51.680022 19792 mini_kdc.cc:121] Time spent starting KDC: real 2.019s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:13:51 dist-test-slave-5nnp krb5kdc[7878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:13:51Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:13:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:13:51.819047 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33213
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:34569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:13:51.899200 7894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:51.899452 7894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:51.899494 7894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:51.900940 7894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:13:51.901010 7894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:51.901032 7894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:51.901053 7894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:13:51.901083 7894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:13:51.902657 7894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33213
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:51.902920 7894 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:51.903110 7894 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:51.905592 7900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:51.905618 7902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:51.905774 7894 server_base.cc:1060] running on GCE node
W20260131 03:13:51.905686 7899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:51.906037 7894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:51.906272 7894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:51.907434 7894 hybrid_clock.cc:648] HybridClock initialized: now 1769829231907421 us; error 25 us; skew 500 ppm
Jan 31 03:13:51 dist-test-slave-5nnp krb5kdc[7878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:13:51.909588 7894 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260131 03:13:51.910089 7894 webserver.cc:492] Webserver started at http://127.0.0.1:40667/ using document root <none> and password file <none>
I20260131 03:13:51.910288 7894 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:51.910343 7894 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:51.910477 7894 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:51.911500 7894 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "ea6bc6da4edd4985b44bf1d3c2366a0b"
format_stamp: "Formatted at 2026-01-31 03:13:51 on dist-test-slave-5nnp"
server_key: "0dc5804c97ee0149b965a9d9acd941a8"
server_key_iv: "8b9214ba6e7bedeafdb2257b304ef54b"
server_key_version: "encryptionkey@0"
I20260131 03:13:51.911818 7894 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "ea6bc6da4edd4985b44bf1d3c2366a0b"
format_stamp: "Formatted at 2026-01-31 03:13:51 on dist-test-slave-5nnp"
server_key: "0dc5804c97ee0149b965a9d9acd941a8"
server_key_iv: "8b9214ba6e7bedeafdb2257b304ef54b"
server_key_version: "encryptionkey@0"
I20260131 03:13:51.912992 7894 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:13:51.913671 7909 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:51.913872 7894 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260131 03:13:51.913959 7894 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "ea6bc6da4edd4985b44bf1d3c2366a0b"
format_stamp: "Formatted at 2026-01-31 03:13:51 on dist-test-slave-5nnp"
server_key: "0dc5804c97ee0149b965a9d9acd941a8"
server_key_iv: "8b9214ba6e7bedeafdb2257b304ef54b"
server_key_version: "encryptionkey@0"
I20260131 03:13:51.914033 7894 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:51.931596 7894 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:51.933538 7894 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:51.933698 7894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:51.937901 7894 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33213
I20260131 03:13:51.937940 7961 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:33213 every 8 connection(s)
I20260131 03:13:51.938271 7894 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:13:51.938901 7962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:51.941519 7962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea6bc6da4edd4985b44bf1d3c2366a0b: Bootstrap starting.
I20260131 03:13:51.942111 7962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea6bc6da4edd4985b44bf1d3c2366a0b: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:51.942384 7962 log.cc:826] T 00000000000000000000000000000000 P ea6bc6da4edd4985b44bf1d3c2366a0b: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:51.943094 7962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea6bc6da4edd4985b44bf1d3c2366a0b: No bootstrap required, opened a new log
I20260131 03:13:51.943636 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 7894
I20260131 03:13:51.943763 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:13:51.943934 19792 external_mini_cluster.cc:1468] Setting key 27efaa66bdc42b63934f83f386f36b82
I20260131 03:13:51.944815 7962 raft_consensus.cc:359] T 00000000000000000000000000000000 P ea6bc6da4edd4985b44bf1d3c2366a0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea6bc6da4edd4985b44bf1d3c2366a0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33213 } }
I20260131 03:13:51.944957 7962 raft_consensus.cc:385] T 00000000000000000000000000000000 P ea6bc6da4edd4985b44bf1d3c2366a0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:51.945012 7962 raft_consensus.cc:740] T 00000000000000000000000000000000 P ea6bc6da4edd4985b44bf1d3c2366a0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6bc6da4edd4985b44bf1d3c2366a0b, State: Initialized, Role: FOLLOWER
I20260131 03:13:51.945159 7962 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea6bc6da4edd4985b44bf1d3c2366a0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea6bc6da4edd4985b44bf1d3c2366a0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33213 } }
I20260131 03:13:51.945235 7962 raft_consensus.cc:399] T 00000000000000000000000000000000 P ea6bc6da4edd4985b44bf1d3c2366a0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:13:51.945257 7962 raft_consensus.cc:493] T 00000000000000000000000000000000 P ea6bc6da4edd4985b44bf1d3c2366a0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:13:51.945309 7962 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ea6bc6da4edd4985b44bf1d3c2366a0b [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:51.945815 7962 raft_consensus.cc:515] T 00000000000000000000000000000000 P ea6bc6da4edd4985b44bf1d3c2366a0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea6bc6da4edd4985b44bf1d3c2366a0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33213 } }
I20260131 03:13:51.945928 7962 leader_election.cc:304] T 00000000000000000000000000000000 P ea6bc6da4edd4985b44bf1d3c2366a0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea6bc6da4edd4985b44bf1d3c2366a0b; no voters:
I20260131 03:13:51.946110 7962 leader_election.cc:290] T 00000000000000000000000000000000 P ea6bc6da4edd4985b44bf1d3c2366a0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:13:51.946142 7967 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ea6bc6da4edd4985b44bf1d3c2366a0b [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:13:51.946259 7967 raft_consensus.cc:697] T 00000000000000000000000000000000 P ea6bc6da4edd4985b44bf1d3c2366a0b [term 1 LEADER]: Becoming Leader. State: Replica: ea6bc6da4edd4985b44bf1d3c2366a0b, State: Running, Role: LEADER
I20260131 03:13:51.946358 7962 sys_catalog.cc:565] T 00000000000000000000000000000000 P ea6bc6da4edd4985b44bf1d3c2366a0b [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:13:51.946408 7967 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea6bc6da4edd4985b44bf1d3c2366a0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea6bc6da4edd4985b44bf1d3c2366a0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33213 } }
I20260131 03:13:51.946743 7967 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea6bc6da4edd4985b44bf1d3c2366a0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea6bc6da4edd4985b44bf1d3c2366a0b. Latest consensus state: current_term: 1 leader_uuid: "ea6bc6da4edd4985b44bf1d3c2366a0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea6bc6da4edd4985b44bf1d3c2366a0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33213 } } }
I20260131 03:13:51.946755 7968 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea6bc6da4edd4985b44bf1d3c2366a0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea6bc6da4edd4985b44bf1d3c2366a0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea6bc6da4edd4985b44bf1d3c2366a0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33213 } } }
I20260131 03:13:51.946962 7968 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea6bc6da4edd4985b44bf1d3c2366a0b [sys.catalog]: This master's current role is: LEADER
I20260131 03:13:51.946885 7967 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea6bc6da4edd4985b44bf1d3c2366a0b [sys.catalog]: This master's current role is: LEADER
I20260131 03:13:51.947827 7975 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:13:51.948324 7975 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:13:51.948949 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:51.944439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57537 (local address 127.0.0.1:33213)
0131 03:13:51.944610 (+ 171us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:51.944621 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:51.944648 (+ 27us) server_negotiation.cc:408] Connection header received
0131 03:13:51.944776 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:51.944780 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:51.944824 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:51.944937 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:51.945345 (+ 408us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:51.946239 (+ 894us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:51.947034 (+ 795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:51.947339 (+ 305us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:51.948282 (+ 943us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:51.948290 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:51.948290 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:13:51.948517 (+ 227us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:51.948541 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:13:51.948550 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:51.948644 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:51.948802 (+ 158us) server_negotiation.cc:300] Negotiation successful
0131 03:13:51.948829 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260131 03:13:51.949976 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:13:51.950089 7975 catalog_manager.cc:1357] Generated new cluster ID: 821bcfee27d64f36a1a84386f27e1876
I20260131 03:13:51.950134 7975 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:13:51.958177 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:51.951645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34623 (local address 127.0.0.1:33213)
0131 03:13:51.951708 (+ 63us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:51.951710 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:51.955612 (+ 3902us) server_negotiation.cc:408] Connection header received
0131 03:13:51.955756 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:51.955757 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:51.955778 (+ 21us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:51.955844 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:51.956128 (+ 284us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:51.956584 (+ 456us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:51.957137 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:51.957317 (+ 180us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:51.957884 (+ 567us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:51.957889 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:51.957889 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:13:51.957894 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:51.957911 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:13:51.957916 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:51.957976 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:51.958055 (+ 79us) server_negotiation.cc:300] Negotiation successful
0131 03:13:51.958067 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260131 03:13:51.969563 7975 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:13:51.970113 7975 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:13:51.976063 7975 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ea6bc6da4edd4985b44bf1d3c2366a0b: Generated new TSK 0
I20260131 03:13:51.976259 7975 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:13:51.979604 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 7894
W20260131 03:13:51.984251 7876 connection.cc:537] client connection to 127.0.0.1:33213 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33213): Connection reset by peer (error 104)
2026-01-31T03:13:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:13:52 dist-test-slave-5nnp krb5kdc[7993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:13:52 dist-test-slave-5nnp krb5kdc[7993](info): set up 2 sockets
Jan 31 03:13:52 dist-test-slave-5nnp krb5kdc[7993](info): commencing operation
krb5kdc: starting...
W20260131 03:13:53.991999 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.993s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:13:54 dist-test-slave-5nnp krb5kdc[7993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:13:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:13:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:13:54.151716 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39395
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:36967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:13:54.232863 8009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:54.233055 8009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:54.233078 8009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:54.234678 8009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:13:54.234727 8009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:54.234742 8009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:54.234754 8009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:13:54.234767 8009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:13:54.236865 8009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39395
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:54.237100 8009 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:54.237324 8009 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:54.239931 8014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:54.239998 8015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:54.239925 8017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:54.240255 8009 server_base.cc:1060] running on GCE node
I20260131 03:13:54.240484 8009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:54.240717 8009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:54.241873 8009 hybrid_clock.cc:648] HybridClock initialized: now 1769829234241865 us; error 49 us; skew 500 ppm
Jan 31 03:13:54 dist-test-slave-5nnp krb5kdc[7993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:13:54.244179 8009 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260131 03:13:54.244673 8009 webserver.cc:492] Webserver started at http://127.0.0.1:45235/ using document root <none> and password file <none>
I20260131 03:13:54.244870 8009 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:54.244907 8009 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:54.245024 8009 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:54.246001 8009 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "c990812c07e44157a5cfb127de8c2a9e"
format_stamp: "Formatted at 2026-01-31 03:13:54 on dist-test-slave-5nnp"
server_key: "bdb7d8753ddfb8167e59b64aaaefcc1f"
server_key_iv: "e0bd49a5d84a096a50d13f670bc0bf71"
server_key_version: "encryptionkey@0"
I20260131 03:13:54.246328 8009 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "c990812c07e44157a5cfb127de8c2a9e"
format_stamp: "Formatted at 2026-01-31 03:13:54 on dist-test-slave-5nnp"
server_key: "bdb7d8753ddfb8167e59b64aaaefcc1f"
server_key_iv: "e0bd49a5d84a096a50d13f670bc0bf71"
server_key_version: "encryptionkey@0"
I20260131 03:13:54.247634 8009 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260131 03:13:54.248399 8024 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:54.248633 8009 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:13:54.248696 8009 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "c990812c07e44157a5cfb127de8c2a9e"
format_stamp: "Formatted at 2026-01-31 03:13:54 on dist-test-slave-5nnp"
server_key: "bdb7d8753ddfb8167e59b64aaaefcc1f"
server_key_iv: "e0bd49a5d84a096a50d13f670bc0bf71"
server_key_version: "encryptionkey@0"
I20260131 03:13:54.248769 8009 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:54.252015 8009 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:54.253808 8009 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:54.253944 8009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:54.258144 8009 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39395
I20260131 03:13:54.258244 8076 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:39395 every 8 connection(s)
I20260131 03:13:54.258524 8009 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:13:54.259155 8077 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:54.261765 8077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c990812c07e44157a5cfb127de8c2a9e: Bootstrap starting.
I20260131 03:13:54.262370 8077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c990812c07e44157a5cfb127de8c2a9e: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:54.262627 8077 log.cc:826] T 00000000000000000000000000000000 P c990812c07e44157a5cfb127de8c2a9e: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:54.263356 8077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c990812c07e44157a5cfb127de8c2a9e: No bootstrap required, opened a new log
I20260131 03:13:54.264587 8077 raft_consensus.cc:359] T 00000000000000000000000000000000 P c990812c07e44157a5cfb127de8c2a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c990812c07e44157a5cfb127de8c2a9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39395 } }
I20260131 03:13:54.264742 8077 raft_consensus.cc:385] T 00000000000000000000000000000000 P c990812c07e44157a5cfb127de8c2a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:54.264786 8077 raft_consensus.cc:740] T 00000000000000000000000000000000 P c990812c07e44157a5cfb127de8c2a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c990812c07e44157a5cfb127de8c2a9e, State: Initialized, Role: FOLLOWER
I20260131 03:13:54.264932 8077 consensus_queue.cc:260] T 00000000000000000000000000000000 P c990812c07e44157a5cfb127de8c2a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c990812c07e44157a5cfb127de8c2a9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39395 } }
I20260131 03:13:54.265035 8077 raft_consensus.cc:399] T 00000000000000000000000000000000 P c990812c07e44157a5cfb127de8c2a9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:13:54.265079 8077 raft_consensus.cc:493] T 00000000000000000000000000000000 P c990812c07e44157a5cfb127de8c2a9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:13:54.265127 8077 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c990812c07e44157a5cfb127de8c2a9e [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:54.265655 8077 raft_consensus.cc:515] T 00000000000000000000000000000000 P c990812c07e44157a5cfb127de8c2a9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c990812c07e44157a5cfb127de8c2a9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39395 } }
I20260131 03:13:54.265784 8077 leader_election.cc:304] T 00000000000000000000000000000000 P c990812c07e44157a5cfb127de8c2a9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c990812c07e44157a5cfb127de8c2a9e; no voters:
I20260131 03:13:54.265940 8077 leader_election.cc:290] T 00000000000000000000000000000000 P c990812c07e44157a5cfb127de8c2a9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:13:54.266049 8080 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c990812c07e44157a5cfb127de8c2a9e [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:13:54.266212 8080 raft_consensus.cc:697] T 00000000000000000000000000000000 P c990812c07e44157a5cfb127de8c2a9e [term 1 LEADER]: Becoming Leader. State: Replica: c990812c07e44157a5cfb127de8c2a9e, State: Running, Role: LEADER
I20260131 03:13:54.266302 8080 consensus_queue.cc:237] T 00000000000000000000000000000000 P c990812c07e44157a5cfb127de8c2a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c990812c07e44157a5cfb127de8c2a9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39395 } }
I20260131 03:13:54.266327 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 8009
I20260131 03:13:54.266470 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:13:54.266656 19792 external_mini_cluster.cc:1468] Setting key 979df25f17f5923c54739c6080c5e635
I20260131 03:13:54.266635 8080 sys_catalog.cc:455] T 00000000000000000000000000000000 P c990812c07e44157a5cfb127de8c2a9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c990812c07e44157a5cfb127de8c2a9e. Latest consensus state: current_term: 1 leader_uuid: "c990812c07e44157a5cfb127de8c2a9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c990812c07e44157a5cfb127de8c2a9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39395 } } }
I20260131 03:13:54.266657 8081 sys_catalog.cc:455] T 00000000000000000000000000000000 P c990812c07e44157a5cfb127de8c2a9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c990812c07e44157a5cfb127de8c2a9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c990812c07e44157a5cfb127de8c2a9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39395 } } }
I20260131 03:13:54.266701 8080 sys_catalog.cc:458] T 00000000000000000000000000000000 P c990812c07e44157a5cfb127de8c2a9e [sys.catalog]: This master's current role is: LEADER
I20260131 03:13:54.266707 8081 sys_catalog.cc:458] T 00000000000000000000000000000000 P c990812c07e44157a5cfb127de8c2a9e [sys.catalog]: This master's current role is: LEADER
I20260131 03:13:54.266947 8077 sys_catalog.cc:565] T 00000000000000000000000000000000 P c990812c07e44157a5cfb127de8c2a9e [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:13:54.267458 8084 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:13:54.267935 8084 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:13:54.269197 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:54.267322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38361 (local address 127.0.0.1:39395)
0131 03:13:54.267629 (+ 307us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:54.267641 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:54.267664 (+ 23us) server_negotiation.cc:408] Connection header received
0131 03:13:54.267778 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:54.267784 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:54.267813 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:54.267946 (+ 133us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:54.268663 (+ 717us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:54.268668 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:54.268668 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:13:54.268672 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:54.268693 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:13:54.268701 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:54.269003 (+ 302us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:54.269040 (+ 37us) server_negotiation.cc:300] Negotiation successful
0131 03:13:54.269074 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":218,"threads_started":1}
I20260131 03:13:54.269956 8084 catalog_manager.cc:1357] Generated new cluster ID: 6b688dffe63a44dc9a746394134930a0
I20260131 03:13:54.270006 8084 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:13:54.270166 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:13:54.278580 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:54.271729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34735 (local address 127.0.0.1:39395)
0131 03:13:54.271838 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:54.271840 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:54.277523 (+ 5683us) server_negotiation.cc:408] Connection header received
0131 03:13:54.277569 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:54.277570 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:54.277576 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:54.277648 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:54.278326 (+ 678us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:54.278328 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:54.278328 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:13:54.278332 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:54.278350 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:13:54.278354 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:54.278418 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:54.278475 (+ 57us) server_negotiation.cc:300] Negotiation successful
0131 03:13:54.278496 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260131 03:13:54.285409 8084 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:13:54.285588 8084 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:13:54.290222 8084 catalog_manager.cc:6027] T 00000000000000000000000000000000 P c990812c07e44157a5cfb127de8c2a9e: Generated new TSK 0
I20260131 03:13:54.290403 8084 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:13:54.292120 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 8009
2026-01-31T03:13:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:13:54 dist-test-slave-5nnp krb5kdc[8108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:13:54 dist-test-slave-5nnp krb5kdc[8108](info): set up 2 sockets
Jan 31 03:13:54 dist-test-slave-5nnp krb5kdc[8108](info): commencing operation
krb5kdc: starting...
W20260131 03:13:56.284014 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.974s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:13:56 dist-test-slave-5nnp krb5kdc[8108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:13:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:13:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:13:56.423996 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:46603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:13:56.501879 8124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:56.502049 8124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:56.502071 8124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:56.503466 8124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:13:56.503504 8124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:56.503518 8124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:56.503530 8124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:13:56.503542 8124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:13:56.505044 8124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:56.505203 8124 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:56.505342 8124 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:56.507704 8130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:56.507715 8132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:56.507803 8129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:56.507848 8124 server_base.cc:1060] running on GCE node
I20260131 03:13:56.508072 8124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:56.508327 8124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:56.509500 8124 hybrid_clock.cc:648] HybridClock initialized: now 1769829236509463 us; error 43 us; skew 500 ppm
Jan 31 03:13:56 dist-test-slave-5nnp krb5kdc[8108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:13:56.511745 8124 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260131 03:13:56.512282 8124 webserver.cc:492] Webserver started at http://127.0.0.1:40801/ using document root <none> and password file <none>
I20260131 03:13:56.512480 8124 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:56.512540 8124 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:56.512676 8124 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:56.513608 8124 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "0fb20cbb7e424ffab66049363aea6eae"
format_stamp: "Formatted at 2026-01-31 03:13:56 on dist-test-slave-5nnp"
server_key: "4c9de33276588d6c6c36b5a147825b7b"
server_key_iv: "9e289cd39315c3e03448ef5eb8548e38"
server_key_version: "encryptionkey@0"
I20260131 03:13:56.513929 8124 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "0fb20cbb7e424ffab66049363aea6eae"
format_stamp: "Formatted at 2026-01-31 03:13:56 on dist-test-slave-5nnp"
server_key: "4c9de33276588d6c6c36b5a147825b7b"
server_key_iv: "9e289cd39315c3e03448ef5eb8548e38"
server_key_version: "encryptionkey@0"
I20260131 03:13:56.515106 8124 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260131 03:13:56.515856 8139 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:56.516024 8124 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:56.516111 8124 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "0fb20cbb7e424ffab66049363aea6eae"
format_stamp: "Formatted at 2026-01-31 03:13:56 on dist-test-slave-5nnp"
server_key: "4c9de33276588d6c6c36b5a147825b7b"
server_key_iv: "9e289cd39315c3e03448ef5eb8548e38"
server_key_version: "encryptionkey@0"
I20260131 03:13:56.516189 8124 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:56.528294 8124 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:56.533058 8124 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:56.533209 8124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:56.537510 8124 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33855
I20260131 03:13:56.537565 8191 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:33855 every 8 connection(s)
I20260131 03:13:56.537889 8124 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:13:56.538220 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 8124
I20260131 03:13:56.538385 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:13:56.538576 19792 external_mini_cluster.cc:1468] Setting key 66b7c9185c72a746461c9f8b6da87151
I20260131 03:13:56.538604 8192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:56.542040 8192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fb20cbb7e424ffab66049363aea6eae: Bootstrap starting.
I20260131 03:13:56.542730 8192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fb20cbb7e424ffab66049363aea6eae: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:56.543015 8192 log.cc:826] T 00000000000000000000000000000000 P 0fb20cbb7e424ffab66049363aea6eae: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:56.543879 8192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fb20cbb7e424ffab66049363aea6eae: No bootstrap required, opened a new log
Jan 31 03:13:56 dist-test-slave-5nnp krb5kdc[8108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260131 03:13:56.545311 8192 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0fb20cbb7e424ffab66049363aea6eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb20cbb7e424ffab66049363aea6eae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33855 } }
I20260131 03:13:56.545459 8192 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0fb20cbb7e424ffab66049363aea6eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:56.545503 8192 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0fb20cbb7e424ffab66049363aea6eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fb20cbb7e424ffab66049363aea6eae, State: Initialized, Role: FOLLOWER
I20260131 03:13:56.545610 8192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fb20cbb7e424ffab66049363aea6eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb20cbb7e424ffab66049363aea6eae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33855 } }
I20260131 03:13:56.545699 8192 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0fb20cbb7e424ffab66049363aea6eae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:13:56.545742 8192 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0fb20cbb7e424ffab66049363aea6eae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:13:56.545785 8192 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0fb20cbb7e424ffab66049363aea6eae [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:56.546368 8192 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0fb20cbb7e424ffab66049363aea6eae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb20cbb7e424ffab66049363aea6eae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33855 } }
I20260131 03:13:56.546489 8192 leader_election.cc:304] T 00000000000000000000000000000000 P 0fb20cbb7e424ffab66049363aea6eae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fb20cbb7e424ffab66049363aea6eae; no voters:
I20260131 03:13:56.546663 8192 leader_election.cc:290] T 00000000000000000000000000000000 P 0fb20cbb7e424ffab66049363aea6eae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:13:56.546859 8197 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0fb20cbb7e424ffab66049363aea6eae [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:13:56.546947 8192 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0fb20cbb7e424ffab66049363aea6eae [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:13:56.547055 8197 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0fb20cbb7e424ffab66049363aea6eae [term 1 LEADER]: Becoming Leader. State: Replica: 0fb20cbb7e424ffab66049363aea6eae, State: Running, Role: LEADER
I20260131 03:13:56.547199 8197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fb20cbb7e424ffab66049363aea6eae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb20cbb7e424ffab66049363aea6eae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33855 } }
I20260131 03:13:56.548086 8198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fb20cbb7e424ffab66049363aea6eae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fb20cbb7e424ffab66049363aea6eae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb20cbb7e424ffab66049363aea6eae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33855 } } }
I20260131 03:13:56.548183 8198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fb20cbb7e424ffab66049363aea6eae [sys.catalog]: This master's current role is: LEADER
I20260131 03:13:56.548592 8199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fb20cbb7e424ffab66049363aea6eae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fb20cbb7e424ffab66049363aea6eae. Latest consensus state: current_term: 1 leader_uuid: "0fb20cbb7e424ffab66049363aea6eae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb20cbb7e424ffab66049363aea6eae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33855 } } }
I20260131 03:13:56.548666 8199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fb20cbb7e424ffab66049363aea6eae [sys.catalog]: This master's current role is: LEADER
W20260131 03:13:56.548841 8212 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 0fb20cbb7e424ffab66049363aea6eae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:13:56.548906 8212 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:13:56.548957 8211 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:13:56.549144 8211 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:13:56.549206 8195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:56.539048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52141 (local address 127.0.0.1:33855)
0131 03:13:56.539215 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:56.539227 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:56.539245 (+ 18us) server_negotiation.cc:408] Connection header received
0131 03:13:56.540014 (+ 769us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:56.540021 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:56.540064 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:56.540165 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:56.541253 (+ 1088us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:56.541963 (+ 710us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:56.542770 (+ 807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:56.543018 (+ 248us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:56.545246 (+ 2228us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:56.545254 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:56.545254 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:56.545283 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:56.547392 (+ 2109us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:56.547747 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:56.547750 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:56.547750 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:56.547818 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:56.547998 (+ 180us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:56.547999 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:56.548000 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:56.548257 (+ 257us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:56.548417 (+ 160us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:56.548888 (+ 471us) server_negotiation.cc:300] Negotiation successful
0131 03:13:56.549034 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260131 03:13:56.550165 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:13:56.552425 8211 catalog_manager.cc:1357] Generated new cluster ID: 7da2dce85c6e44689a3175aa5a3d2fb8
I20260131 03:13:56.552476 8211 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:13:56.558506 8195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:56.551533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53057 (local address 127.0.0.1:33855)
0131 03:13:56.551596 (+ 63us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:56.551598 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:56.551748 (+ 150us) server_negotiation.cc:408] Connection header received
0131 03:13:56.551797 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:56.551798 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:56.551810 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:56.551865 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:56.552719 (+ 854us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:56.553403 (+ 684us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:56.554171 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:56.554361 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:56.555798 (+ 1437us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:56.555803 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:56.555803 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:13:56.555810 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:56.557365 (+ 1555us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:56.557744 (+ 379us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:56.557745 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:56.557746 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:56.557776 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:13:56.557941 (+ 165us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:13:56.557942 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:13:56.557942 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:13:56.558046 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:56.558116 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:56.558316 (+ 200us) server_negotiation.cc:300] Negotiation successful
0131 03:13:56.558398 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260131 03:13:56.560729 8211 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:13:56.561266 8211 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:13:56.569437 8211 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 0fb20cbb7e424ffab66049363aea6eae: Generated new TSK 0
I20260131 03:13:56.569655 8211 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:13:56.572638 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 8124
2026-01-31T03:13:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2280 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:13:56 dist-test-slave-5nnp krb5kdc[8224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:13:56 dist-test-slave-5nnp krb5kdc[8224](info): set up 2 sockets
Jan 31 03:13:56 dist-test-slave-5nnp krb5kdc[8224](info): commencing operation
krb5kdc: starting...
W20260131 03:13:58.572059 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.982s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:13:58 dist-test-slave-5nnp krb5kdc[8224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:13:58Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:13:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:13:58.716475 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43173
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:37453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:13:58.796627 8240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:13:58.796869 8240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:13:58.796911 8240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:13:58.798444 8240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:13:58.798535 8240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:13:58.798563 8240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:13:58.798596 8240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:13:58.798625 8240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:13:58.800467 8240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:13:58.800736 8240 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:13:58.800947 8240 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:13:58.803496 8248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:58.803629 8246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:13:58.803496 8245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:13:58.803695 8240 server_base.cc:1060] running on GCE node
I20260131 03:13:58.804019 8240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:13:58.804327 8240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:13:58.805501 8240 hybrid_clock.cc:648] HybridClock initialized: now 1769829238805470 us; error 51 us; skew 500 ppm
Jan 31 03:13:58 dist-test-slave-5nnp krb5kdc[8224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:13:58.808059 8240 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260131 03:13:58.808638 8240 webserver.cc:492] Webserver started at http://127.0.0.1:33519/ using document root <none> and password file <none>
I20260131 03:13:58.808845 8240 fs_manager.cc:362] Metadata directory not provided
I20260131 03:13:58.808900 8240 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:13:58.809039 8240 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:13:58.810019 8240 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "9a28da9b5322456f8f64f42d14856a83"
format_stamp: "Formatted at 2026-01-31 03:13:58 on dist-test-slave-5nnp"
server_key: "6e791508aa17164618cd6c1c9d99e2b6"
server_key_iv: "e13762b126c3821db574736359e2b95b"
server_key_version: "encryptionkey@0"
I20260131 03:13:58.810389 8240 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "9a28da9b5322456f8f64f42d14856a83"
format_stamp: "Formatted at 2026-01-31 03:13:58 on dist-test-slave-5nnp"
server_key: "6e791508aa17164618cd6c1c9d99e2b6"
server_key_iv: "e13762b126c3821db574736359e2b95b"
server_key_version: "encryptionkey@0"
I20260131 03:13:58.811750 8240 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:13:58.812544 8255 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:13:58.812774 8240 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:13:58.812875 8240 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "9a28da9b5322456f8f64f42d14856a83"
format_stamp: "Formatted at 2026-01-31 03:13:58 on dist-test-slave-5nnp"
server_key: "6e791508aa17164618cd6c1c9d99e2b6"
server_key_iv: "e13762b126c3821db574736359e2b95b"
server_key_version: "encryptionkey@0"
I20260131 03:13:58.812958 8240 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:13:58.826630 8240 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:13:58.829041 8240 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:13:58.829200 8240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:13:58.833508 8240 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43173
I20260131 03:13:58.833563 8307 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:43173 every 8 connection(s)
I20260131 03:13:58.833902 8240 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:13:58.834597 8308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:13:58.837177 8308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a28da9b5322456f8f64f42d14856a83: Bootstrap starting.
I20260131 03:13:58.837728 8308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a28da9b5322456f8f64f42d14856a83: Neither blocks nor log segments found. Creating new log.
I20260131 03:13:58.837932 8308 log.cc:826] T 00000000000000000000000000000000 P 9a28da9b5322456f8f64f42d14856a83: Log is configured to *not* fsync() on all Append() calls
I20260131 03:13:58.838688 8308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a28da9b5322456f8f64f42d14856a83: No bootstrap required, opened a new log
I20260131 03:13:58.839967 8308 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9a28da9b5322456f8f64f42d14856a83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a28da9b5322456f8f64f42d14856a83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43173 } }
I20260131 03:13:58.840094 8308 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9a28da9b5322456f8f64f42d14856a83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:13:58.840116 8308 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9a28da9b5322456f8f64f42d14856a83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a28da9b5322456f8f64f42d14856a83, State: Initialized, Role: FOLLOWER
I20260131 03:13:58.840242 8308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a28da9b5322456f8f64f42d14856a83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a28da9b5322456f8f64f42d14856a83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43173 } }
I20260131 03:13:58.840332 8308 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9a28da9b5322456f8f64f42d14856a83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:13:58.840356 8308 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9a28da9b5322456f8f64f42d14856a83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:13:58.840384 8308 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9a28da9b5322456f8f64f42d14856a83 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:13:58.840876 8308 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9a28da9b5322456f8f64f42d14856a83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a28da9b5322456f8f64f42d14856a83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43173 } }
I20260131 03:13:58.840999 8308 leader_election.cc:304] T 00000000000000000000000000000000 P 9a28da9b5322456f8f64f42d14856a83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a28da9b5322456f8f64f42d14856a83; no voters:
I20260131 03:13:58.841158 8308 leader_election.cc:290] T 00000000000000000000000000000000 P 9a28da9b5322456f8f64f42d14856a83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:13:58.841181 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 8240
I20260131 03:13:58.841300 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:13:58.841264 8311 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9a28da9b5322456f8f64f42d14856a83 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:13:58.841444 8311 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9a28da9b5322456f8f64f42d14856a83 [term 1 LEADER]: Becoming Leader. State: Replica: 9a28da9b5322456f8f64f42d14856a83, State: Running, Role: LEADER
I20260131 03:13:58.841487 19792 external_mini_cluster.cc:1468] Setting key 44533f22803d3c6c32e74636b7b3c89c
I20260131 03:13:58.841450 8308 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9a28da9b5322456f8f64f42d14856a83 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:13:58.841650 8311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a28da9b5322456f8f64f42d14856a83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a28da9b5322456f8f64f42d14856a83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43173 } }
I20260131 03:13:58.842056 8312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a28da9b5322456f8f64f42d14856a83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a28da9b5322456f8f64f42d14856a83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a28da9b5322456f8f64f42d14856a83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43173 } } }
I20260131 03:13:58.842137 8312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a28da9b5322456f8f64f42d14856a83 [sys.catalog]: This master's current role is: LEADER
I20260131 03:13:58.842384 8313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a28da9b5322456f8f64f42d14856a83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a28da9b5322456f8f64f42d14856a83. Latest consensus state: current_term: 1 leader_uuid: "9a28da9b5322456f8f64f42d14856a83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a28da9b5322456f8f64f42d14856a83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43173 } } }
I20260131 03:13:58.842468 8313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a28da9b5322456f8f64f42d14856a83 [sys.catalog]: This master's current role is: LEADER
I20260131 03:13:58.842660 8320 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:13:58.843434 8320 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:13:58.844969 8320 catalog_manager.cc:1357] Generated new cluster ID: 58026552ac624996aa01d6cbdae18d61
I20260131 03:13:58.845024 8320 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:13:58.846622 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:58.842051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37933 (local address 127.0.0.1:43173)
0131 03:13:58.842210 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:58.842221 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:58.842243 (+ 22us) server_negotiation.cc:408] Connection header received
0131 03:13:58.842389 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:58.842395 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:58.842446 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:58.843118 (+ 672us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:58.843269 (+ 151us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:58.844482 (+ 1213us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:58.845246 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:58.845507 (+ 261us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:58.845983 (+ 476us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:58.846006 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:58.846006 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:13:58.846199 (+ 193us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:58.846221 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:13:58.846228 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:58.846338 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:58.846425 (+ 87us) server_negotiation.cc:300] Negotiation successful
0131 03:13:58.846469 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260131 03:13:58.847509 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:13:58.854252 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:13:58.848868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39351 (local address 127.0.0.1:43173)
0131 03:13:58.848908 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0131 03:13:58.848910 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:13:58.851610 (+ 2700us) server_negotiation.cc:408] Connection header received
0131 03:13:58.851708 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:13:58.851708 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:13:58.851727 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:13:58.851790 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:13:58.852205 (+ 415us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:58.852690 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:13:58.853232 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:13:58.853427 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:13:58.853936 (+ 509us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:13:58.853939 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:13:58.853940 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:13:58.853946 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:13:58.853962 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:13:58.853971 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:13:58.854026 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0131 03:13:58.854111 (+ 85us) server_negotiation.cc:300] Negotiation successful
0131 03:13:58.854142 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260131 03:13:58.862970 8320 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:13:58.863766 8320 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:13:58.870901 8320 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 9a28da9b5322456f8f64f42d14856a83: Generated new TSK 0
I20260131 03:13:58.871132 8320 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:13:58.873337 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 8240
2026-01-31T03:13:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:13:58 dist-test-slave-5nnp krb5kdc[8339](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:13:58 dist-test-slave-5nnp krb5kdc[8339](info): set up 2 sockets
Jan 31 03:13:58 dist-test-slave-5nnp krb5kdc[8339](info): commencing operation
krb5kdc: starting...
W20260131 03:14:00.868155 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.977s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:14:00 dist-test-slave-5nnp krb5kdc[8339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:14:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:14:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:14:01.006369 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43383
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:46489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:14:01.085731 8355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:14:01.085922 8355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:14:01.085945 8355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:14:01.087510 8355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:14:01.087563 8355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:14:01.087579 8355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:14:01.087590 8355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:14:01.087602 8355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:14:01.089318 8355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:14:01.089567 8355 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:14:01.089787 8355 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:14:01.092339 8363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:01.092509 8360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:01.092415 8361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:01.092597 8355 server_base.cc:1060] running on GCE node
I20260131 03:14:01.092923 8355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:14:01.093149 8355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:14:01.094301 8355 hybrid_clock.cc:648] HybridClock initialized: now 1769829241094271 us; error 53 us; skew 500 ppm
Jan 31 03:14:01 dist-test-slave-5nnp krb5kdc[8339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:14:01.096753 8355 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260131 03:14:01.097295 8355 webserver.cc:492] Webserver started at http://127.0.0.1:35593/ using document root <none> and password file <none>
I20260131 03:14:01.097499 8355 fs_manager.cc:362] Metadata directory not provided
I20260131 03:14:01.097556 8355 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:14:01.097702 8355 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:14:01.098675 8355 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "9d452bf8ba2144c084286e7978aada32"
format_stamp: "Formatted at 2026-01-31 03:14:01 on dist-test-slave-5nnp"
server_key: "ec25e2872a5a5c98e6efc1ea4179f967"
server_key_iv: "d911023b353079a17a46c580b5517f1a"
server_key_version: "encryptionkey@0"
I20260131 03:14:01.099006 8355 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "9d452bf8ba2144c084286e7978aada32"
format_stamp: "Formatted at 2026-01-31 03:14:01 on dist-test-slave-5nnp"
server_key: "ec25e2872a5a5c98e6efc1ea4179f967"
server_key_iv: "d911023b353079a17a46c580b5517f1a"
server_key_version: "encryptionkey@0"
I20260131 03:14:01.100311 8355 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260131 03:14:01.101073 8370 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:14:01.101265 8355 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:14:01.101352 8355 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "9d452bf8ba2144c084286e7978aada32"
format_stamp: "Formatted at 2026-01-31 03:14:01 on dist-test-slave-5nnp"
server_key: "ec25e2872a5a5c98e6efc1ea4179f967"
server_key_iv: "d911023b353079a17a46c580b5517f1a"
server_key_version: "encryptionkey@0"
I20260131 03:14:01.101437 8355 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:14:01.105113 8355 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:14:01.107205 8355 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:14:01.107358 8355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:14:01.111539 8355 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43383
I20260131 03:14:01.111588 8423 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:43383 every 8 connection(s)
I20260131 03:14:01.111898 8355 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:14:01.112494 8424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:14:01.115015 8424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d452bf8ba2144c084286e7978aada32: Bootstrap starting.
I20260131 03:14:01.115700 8424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d452bf8ba2144c084286e7978aada32: Neither blocks nor log segments found. Creating new log.
I20260131 03:14:01.115959 8424 log.cc:826] T 00000000000000000000000000000000 P 9d452bf8ba2144c084286e7978aada32: Log is configured to *not* fsync() on all Append() calls
I20260131 03:14:01.116704 8424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d452bf8ba2144c084286e7978aada32: No bootstrap required, opened a new log
I20260131 03:14:01.117889 8424 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9d452bf8ba2144c084286e7978aada32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d452bf8ba2144c084286e7978aada32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43383 } }
I20260131 03:14:01.118012 8424 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9d452bf8ba2144c084286e7978aada32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:14:01.118031 8424 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9d452bf8ba2144c084286e7978aada32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d452bf8ba2144c084286e7978aada32, State: Initialized, Role: FOLLOWER
I20260131 03:14:01.118175 8424 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d452bf8ba2144c084286e7978aada32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d452bf8ba2144c084286e7978aada32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43383 } }
I20260131 03:14:01.118233 8424 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9d452bf8ba2144c084286e7978aada32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:14:01.118255 8424 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9d452bf8ba2144c084286e7978aada32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:14:01.118284 8424 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9d452bf8ba2144c084286e7978aada32 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:14:01.118842 8424 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9d452bf8ba2144c084286e7978aada32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d452bf8ba2144c084286e7978aada32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43383 } }
I20260131 03:14:01.118975 8424 leader_election.cc:304] T 00000000000000000000000000000000 P 9d452bf8ba2144c084286e7978aada32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d452bf8ba2144c084286e7978aada32; no voters:
I20260131 03:14:01.119136 8424 leader_election.cc:290] T 00000000000000000000000000000000 P 9d452bf8ba2144c084286e7978aada32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:14:01.119235 8427 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9d452bf8ba2144c084286e7978aada32 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:14:01.119401 8427 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9d452bf8ba2144c084286e7978aada32 [term 1 LEADER]: Becoming Leader. State: Replica: 9d452bf8ba2144c084286e7978aada32, State: Running, Role: LEADER
I20260131 03:14:01.119452 8424 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9d452bf8ba2144c084286e7978aada32 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:14:01.119547 8427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d452bf8ba2144c084286e7978aada32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d452bf8ba2144c084286e7978aada32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43383 } }
I20260131 03:14:01.119940 8429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d452bf8ba2144c084286e7978aada32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d452bf8ba2144c084286e7978aada32. Latest consensus state: current_term: 1 leader_uuid: "9d452bf8ba2144c084286e7978aada32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d452bf8ba2144c084286e7978aada32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43383 } } }
I20260131 03:14:01.119927 8428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d452bf8ba2144c084286e7978aada32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d452bf8ba2144c084286e7978aada32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d452bf8ba2144c084286e7978aada32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43383 } } }
I20260131 03:14:01.120086 8428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d452bf8ba2144c084286e7978aada32 [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:01.120078 8429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d452bf8ba2144c084286e7978aada32 [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:01.120303 8434 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:14:01.120750 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 8355
I20260131 03:14:01.120790 8434 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:14:01.120873 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:14:01.121048 19792 external_mini_cluster.cc:1468] Setting key c60fc8ad007076b2ccc5ebc06b53d34d
I20260131 03:14:01.123000 8434 catalog_manager.cc:1357] Generated new cluster ID: d902e5d9ae2c4cc2a2de88105827f52d
I20260131 03:14:01.123073 8434 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:14:01.123453 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:01.121667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55507 (local address 127.0.0.1:43383)
0131 03:14:01.121843 (+ 176us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:01.121854 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:01.121880 (+ 26us) server_negotiation.cc:408] Connection header received
0131 03:14:01.121984 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:01.121990 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:01.122019 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:01.122147 (+ 128us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:01.122882 (+ 735us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:01.122884 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:01.122885 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:14:01.122889 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:01.122913 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:14:01.122920 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:01.123013 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:01.123236 (+ 223us) server_negotiation.cc:300] Negotiation successful
0131 03:14:01.123270 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20260131 03:14:01.124359 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:14:01.136729 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:01.127661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54083 (local address 127.0.0.1:43383)
0131 03:14:01.127697 (+ 36us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:01.127699 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:01.135588 (+ 7889us) server_negotiation.cc:408] Connection header received
0131 03:14:01.135696 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:01.135698 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:01.135704 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:01.135766 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:01.136484 (+ 718us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:01.136485 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:01.136486 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:14:01.136489 (+ 3us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:01.136506 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:14:01.136510 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:01.136563 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:01.136638 (+ 75us) server_negotiation.cc:300] Negotiation successful
0131 03:14:01.136653 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260131 03:14:01.150619 8434 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:14:01.150815 8434 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:14:01.159145 8434 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 9d452bf8ba2144c084286e7978aada32: Generated new TSK 0
I20260131 03:14:01.159390 8434 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:14:01.170423 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 8355
2026-01-31T03:14:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:14:01 dist-test-slave-5nnp krb5kdc[8455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:14:01 dist-test-slave-5nnp krb5kdc[8455](info): set up 2 sockets
Jan 31 03:14:01 dist-test-slave-5nnp krb5kdc[8455](info): commencing operation
krb5kdc: starting...
W20260131 03:14:03.183944 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.995s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:14:03 dist-test-slave-5nnp krb5kdc[8455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:14:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:14:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:14:03.322026 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40557
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:43217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:14:03.399955 8471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:14:03.400130 8471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:14:03.400151 8471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:14:03.401692 8471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:14:03.401738 8471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:14:03.401753 8471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:14:03.401767 8471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:14:03.401815 8471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:14:03.403474 8471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:14:03.403734 8471 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:14:03.403928 8471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:14:03.406342 8477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:03.406337 8476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:03.406332 8479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:03.406507 8471 server_base.cc:1060] running on GCE node
I20260131 03:14:03.406839 8471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:14:03.407094 8471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:14:03.408243 8471 hybrid_clock.cc:648] HybridClock initialized: now 1769829243408237 us; error 49 us; skew 500 ppm
Jan 31 03:14:03 dist-test-slave-5nnp krb5kdc[8455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:14:03.410418 8471 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260131 03:14:03.410993 8471 webserver.cc:492] Webserver started at http://127.0.0.1:40429/ using document root <none> and password file <none>
I20260131 03:14:03.411191 8471 fs_manager.cc:362] Metadata directory not provided
I20260131 03:14:03.411247 8471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:14:03.411427 8471 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:14:03.412377 8471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "240eea107b7e4c0a91f27b297be54791"
format_stamp: "Formatted at 2026-01-31 03:14:03 on dist-test-slave-5nnp"
server_key: "54c8d880cd3414df5980b955f5287fda"
server_key_iv: "0d35a9d843c9ac8e7d8212409c8c9fff"
server_key_version: "encryptionkey@0"
I20260131 03:14:03.412709 8471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "240eea107b7e4c0a91f27b297be54791"
format_stamp: "Formatted at 2026-01-31 03:14:03 on dist-test-slave-5nnp"
server_key: "54c8d880cd3414df5980b955f5287fda"
server_key_iv: "0d35a9d843c9ac8e7d8212409c8c9fff"
server_key_version: "encryptionkey@0"
I20260131 03:14:03.413883 8471 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260131 03:14:03.414563 8486 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:14:03.414793 8471 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:14:03.414880 8471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "240eea107b7e4c0a91f27b297be54791"
format_stamp: "Formatted at 2026-01-31 03:14:03 on dist-test-slave-5nnp"
server_key: "54c8d880cd3414df5980b955f5287fda"
server_key_iv: "0d35a9d843c9ac8e7d8212409c8c9fff"
server_key_version: "encryptionkey@0"
I20260131 03:14:03.414955 8471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:14:03.430663 8471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:14:03.432850 8471 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:14:03.433009 8471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:14:03.437084 8471 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40557
I20260131 03:14:03.437116 8538 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:40557 every 8 connection(s)
I20260131 03:14:03.437440 8471 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:14:03.438028 8539 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:14:03.440533 8539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 240eea107b7e4c0a91f27b297be54791: Bootstrap starting.
I20260131 03:14:03.441097 8539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 240eea107b7e4c0a91f27b297be54791: Neither blocks nor log segments found. Creating new log.
I20260131 03:14:03.441345 8539 log.cc:826] T 00000000000000000000000000000000 P 240eea107b7e4c0a91f27b297be54791: Log is configured to *not* fsync() on all Append() calls
I20260131 03:14:03.442011 8539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 240eea107b7e4c0a91f27b297be54791: No bootstrap required, opened a new log
I20260131 03:14:03.443202 8539 raft_consensus.cc:359] T 00000000000000000000000000000000 P 240eea107b7e4c0a91f27b297be54791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240eea107b7e4c0a91f27b297be54791" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40557 } }
I20260131 03:14:03.443320 8539 raft_consensus.cc:385] T 00000000000000000000000000000000 P 240eea107b7e4c0a91f27b297be54791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:14:03.443341 8539 raft_consensus.cc:740] T 00000000000000000000000000000000 P 240eea107b7e4c0a91f27b297be54791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 240eea107b7e4c0a91f27b297be54791, State: Initialized, Role: FOLLOWER
I20260131 03:14:03.443509 8539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 240eea107b7e4c0a91f27b297be54791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240eea107b7e4c0a91f27b297be54791" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40557 } }
I20260131 03:14:03.443568 8539 raft_consensus.cc:399] T 00000000000000000000000000000000 P 240eea107b7e4c0a91f27b297be54791 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:14:03.443590 8539 raft_consensus.cc:493] T 00000000000000000000000000000000 P 240eea107b7e4c0a91f27b297be54791 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:14:03.443643 8539 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 240eea107b7e4c0a91f27b297be54791 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:14:03.444151 8539 raft_consensus.cc:515] T 00000000000000000000000000000000 P 240eea107b7e4c0a91f27b297be54791 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240eea107b7e4c0a91f27b297be54791" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40557 } }
I20260131 03:14:03.444262 8539 leader_election.cc:304] T 00000000000000000000000000000000 P 240eea107b7e4c0a91f27b297be54791 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 240eea107b7e4c0a91f27b297be54791; no voters:
I20260131 03:14:03.444429 8539 leader_election.cc:290] T 00000000000000000000000000000000 P 240eea107b7e4c0a91f27b297be54791 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:14:03.444511 8542 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 240eea107b7e4c0a91f27b297be54791 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:14:03.444648 8539 sys_catalog.cc:565] T 00000000000000000000000000000000 P 240eea107b7e4c0a91f27b297be54791 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:14:03.444654 8542 raft_consensus.cc:697] T 00000000000000000000000000000000 P 240eea107b7e4c0a91f27b297be54791 [term 1 LEADER]: Becoming Leader. State: Replica: 240eea107b7e4c0a91f27b297be54791, State: Running, Role: LEADER
I20260131 03:14:03.444769 8542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 240eea107b7e4c0a91f27b297be54791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240eea107b7e4c0a91f27b297be54791" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40557 } }
I20260131 03:14:03.445186 8544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 240eea107b7e4c0a91f27b297be54791 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 240eea107b7e4c0a91f27b297be54791. Latest consensus state: current_term: 1 leader_uuid: "240eea107b7e4c0a91f27b297be54791" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240eea107b7e4c0a91f27b297be54791" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40557 } } }
I20260131 03:14:03.445272 8544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 240eea107b7e4c0a91f27b297be54791 [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:03.445465 8543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 240eea107b7e4c0a91f27b297be54791 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "240eea107b7e4c0a91f27b297be54791" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240eea107b7e4c0a91f27b297be54791" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40557 } } }
I20260131 03:14:03.445536 8543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 240eea107b7e4c0a91f27b297be54791 [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:03.445689 8551 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:14:03.446424 8551 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:14:03.446413 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 8471
I20260131 03:14:03.446537 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:14:03.446738 19792 external_mini_cluster.cc:1468] Setting key 7ee2f2aae71e3ef573aa937fdf0255f0
I20260131 03:14:03.448288 8551 catalog_manager.cc:1357] Generated new cluster ID: 99ce5ca274984e6cace4393267243964
I20260131 03:14:03.448338 8551 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Jan 31 03:14:03 dist-test-slave-5nnp krb5kdc[8455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260131 03:14:03.455513 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:03.447181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:46495 (local address 127.0.0.1:40557)
0131 03:14:03.447475 (+ 294us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:03.447485 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:03.447506 (+ 21us) server_negotiation.cc:408] Connection header received
0131 03:14:03.447611 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:03.447617 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:03.447673 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:03.447825 (+ 152us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:03.448729 (+ 904us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:03.449482 (+ 753us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:03.450045 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:03.450304 (+ 259us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:03.452336 (+ 2032us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:03.452342 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:03.452343 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:03.452364 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:03.454032 (+ 1668us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:03.454422 (+ 390us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:03.454424 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:03.454425 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:03.454492 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:03.454698 (+ 206us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:03.454700 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:03.454700 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:03.454898 (+ 198us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:03.455086 (+ 188us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:03.455183 (+ 97us) server_negotiation.cc:300] Negotiation successful
0131 03:14:03.455296 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":69,"threads_started":1}
I20260131 03:14:03.456418 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:14:03.469584 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:03.457942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:35853 (local address 127.0.0.1:40557)
0131 03:14:03.457980 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:03.457983 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:03.463655 (+ 5672us) server_negotiation.cc:408] Connection header received
0131 03:14:03.463886 (+ 231us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:03.463888 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:03.463902 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:03.463974 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:03.464727 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:03.465184 (+ 457us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:03.465747 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:03.466273 (+ 526us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:03.466989 (+ 716us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:03.466993 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:03.466993 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:03.467003 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:03.468451 (+ 1448us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:03.468812 (+ 361us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:03.468813 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:03.468814 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:03.468843 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:03.469031 (+ 188us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:03.469032 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:03.469032 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:03.469118 (+ 86us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:03.469234 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:03.469360 (+ 126us) server_negotiation.cc:300] Negotiation successful
0131 03:14:03.469428 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260131 03:14:03.484237 8551 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:14:03.485013 8551 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:14:03.492290 8551 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 240eea107b7e4c0a91f27b297be54791: Generated new TSK 0
I20260131 03:14:03.492488 8551 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:14:03.502594 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 8471
W20260131 03:14:03.507057 8453 connection.cc:537] client connection to 127.0.0.1:40557 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40557): Connection reset by peer (error 104)
2026-01-31T03:14:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:14:03 dist-test-slave-5nnp krb5kdc[8570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:14:03 dist-test-slave-5nnp krb5kdc[8570](info): set up 2 sockets
Jan 31 03:14:03 dist-test-slave-5nnp krb5kdc[8570](info): commencing operation
krb5kdc: starting...
W20260131 03:14:05.500236 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.979s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:14:05 dist-test-slave-5nnp krb5kdc[8570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:14:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:14:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:14:05.638978 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41325
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:43651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:14:05.719223 8586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:14:05.719492 8586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:14:05.719538 8586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:14:05.721051 8586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:14:05.721138 8586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:14:05.721163 8586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:14:05.721184 8586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:14:05.721215 8586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:14:05.722884 8586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:14:05.723167 8586 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:14:05.723399 8586 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:14:05.726293 8592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:05.726351 8591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:05.726293 8594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:05.726346 8586 server_base.cc:1060] running on GCE node
I20260131 03:14:05.726835 8586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:14:05.727118 8586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:14:05.728287 8586 hybrid_clock.cc:648] HybridClock initialized: now 1769829245728248 us; error 55 us; skew 500 ppm
Jan 31 03:14:05 dist-test-slave-5nnp krb5kdc[8570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:14:05.731041 8586 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260131 03:14:05.731597 8586 webserver.cc:492] Webserver started at http://127.0.0.1:43545/ using document root <none> and password file <none>
I20260131 03:14:05.731812 8586 fs_manager.cc:362] Metadata directory not provided
I20260131 03:14:05.731850 8586 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:14:05.731969 8586 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:14:05.732986 8586 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "2f45019170374cd6a1f83d3e6639521f"
format_stamp: "Formatted at 2026-01-31 03:14:05 on dist-test-slave-5nnp"
server_key: "6ad698212562309a5691ca451da5e712"
server_key_iv: "c41accdccee9010be573a225085ebd9d"
server_key_version: "encryptionkey@0"
I20260131 03:14:05.733312 8586 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "2f45019170374cd6a1f83d3e6639521f"
format_stamp: "Formatted at 2026-01-31 03:14:05 on dist-test-slave-5nnp"
server_key: "6ad698212562309a5691ca451da5e712"
server_key_iv: "c41accdccee9010be573a225085ebd9d"
server_key_version: "encryptionkey@0"
I20260131 03:14:05.734586 8586 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:14:05.735467 8601 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:14:05.735690 8586 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:14:05.735792 8586 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "2f45019170374cd6a1f83d3e6639521f"
format_stamp: "Formatted at 2026-01-31 03:14:05 on dist-test-slave-5nnp"
server_key: "6ad698212562309a5691ca451da5e712"
server_key_iv: "c41accdccee9010be573a225085ebd9d"
server_key_version: "encryptionkey@0"
I20260131 03:14:05.735880 8586 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:14:05.748970 8586 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:14:05.751327 8586 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:14:05.751528 8586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:14:05.755702 8586 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41325
I20260131 03:14:05.755750 8653 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:41325 every 8 connection(s)
I20260131 03:14:05.756080 8586 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:14:05.756737 8654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:14:05.759248 8654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f45019170374cd6a1f83d3e6639521f: Bootstrap starting.
I20260131 03:14:05.759886 8654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f45019170374cd6a1f83d3e6639521f: Neither blocks nor log segments found. Creating new log.
I20260131 03:14:05.760155 8654 log.cc:826] T 00000000000000000000000000000000 P 2f45019170374cd6a1f83d3e6639521f: Log is configured to *not* fsync() on all Append() calls
I20260131 03:14:05.760943 8654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f45019170374cd6a1f83d3e6639521f: No bootstrap required, opened a new log
I20260131 03:14:05.762181 8654 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2f45019170374cd6a1f83d3e6639521f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f45019170374cd6a1f83d3e6639521f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41325 } }
I20260131 03:14:05.762305 8654 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2f45019170374cd6a1f83d3e6639521f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:14:05.762326 8654 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2f45019170374cd6a1f83d3e6639521f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f45019170374cd6a1f83d3e6639521f, State: Initialized, Role: FOLLOWER
I20260131 03:14:05.762504 8654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f45019170374cd6a1f83d3e6639521f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f45019170374cd6a1f83d3e6639521f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41325 } }
I20260131 03:14:05.762589 8654 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2f45019170374cd6a1f83d3e6639521f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:14:05.762643 8654 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2f45019170374cd6a1f83d3e6639521f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:14:05.762703 8654 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2f45019170374cd6a1f83d3e6639521f [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:14:05.763269 8654 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2f45019170374cd6a1f83d3e6639521f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f45019170374cd6a1f83d3e6639521f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41325 } }
I20260131 03:14:05.763278 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 8586
I20260131 03:14:05.763363 8654 leader_election.cc:304] T 00000000000000000000000000000000 P 2f45019170374cd6a1f83d3e6639521f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f45019170374cd6a1f83d3e6639521f; no voters:
I20260131 03:14:05.763455 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:14:05.763612 8654 leader_election.cc:290] T 00000000000000000000000000000000 P 2f45019170374cd6a1f83d3e6639521f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:14:05.763660 8657 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2f45019170374cd6a1f83d3e6639521f [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:14:05.763715 19792 external_mini_cluster.cc:1468] Setting key 40fcb20b0f481ab07cbbe06f378fcd38
I20260131 03:14:05.763873 8654 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2f45019170374cd6a1f83d3e6639521f [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:14:05.763878 8657 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2f45019170374cd6a1f83d3e6639521f [term 1 LEADER]: Becoming Leader. State: Replica: 2f45019170374cd6a1f83d3e6639521f, State: Running, Role: LEADER
I20260131 03:14:05.764151 8657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f45019170374cd6a1f83d3e6639521f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f45019170374cd6a1f83d3e6639521f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41325 } }
I20260131 03:14:05.764577 8658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f45019170374cd6a1f83d3e6639521f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f45019170374cd6a1f83d3e6639521f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f45019170374cd6a1f83d3e6639521f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41325 } } }
I20260131 03:14:05.764673 8658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f45019170374cd6a1f83d3e6639521f [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:05.765352 8659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f45019170374cd6a1f83d3e6639521f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f45019170374cd6a1f83d3e6639521f. Latest consensus state: current_term: 1 leader_uuid: "2f45019170374cd6a1f83d3e6639521f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f45019170374cd6a1f83d3e6639521f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41325 } } }
I20260131 03:14:05.765524 8659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f45019170374cd6a1f83d3e6639521f [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:05.765741 8669 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:14:05.766204 8669 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:14:05.767776 8669 catalog_manager.cc:1357] Generated new cluster ID: 4de82546bfeb4f7eb6609c3ff511d73e
I20260131 03:14:05.767832 8669 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:14:05.769440 8661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:05.764308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:57981 (local address 127.0.0.1:41325)
0131 03:14:05.764579 (+ 271us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:05.764592 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:05.764617 (+ 25us) server_negotiation.cc:408] Connection header received
0131 03:14:05.764916 (+ 299us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:05.764926 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:05.764994 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:05.765144 (+ 150us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:05.765669 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:05.767123 (+ 1454us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:05.768155 (+ 1032us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:05.768381 (+ 226us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:05.768756 (+ 375us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:05.768781 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:05.768781 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:14:05.769002 (+ 221us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:05.769026 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:14:05.769034 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:05.769140 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:05.769257 (+ 117us) server_negotiation.cc:300] Negotiation successful
0131 03:14:05.769283 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":68,"threads_started":1}
I20260131 03:14:05.770388 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:14:05.775224 8661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:05.771857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:34231 (local address 127.0.0.1:41325)
0131 03:14:05.771890 (+ 33us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:05.771891 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:05.771999 (+ 108us) server_negotiation.cc:408] Connection header received
0131 03:14:05.772089 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:05.772090 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:05.772100 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:05.772128 (+ 28us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:05.772510 (+ 382us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:05.773190 (+ 680us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:05.773960 (+ 770us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:05.774129 (+ 169us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:05.774875 (+ 746us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:05.774878 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:05.774878 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:14:05.774883 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:05.774897 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:14:05.774901 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:05.774936 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:05.775139 (+ 203us) server_negotiation.cc:300] Negotiation successful
0131 03:14:05.775149 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260131 03:14:05.784209 8669 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:14:05.784771 8669 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:14:05.790225 8669 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 2f45019170374cd6a1f83d3e6639521f: Generated new TSK 0
I20260131 03:14:05.790402 8669 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:14:05.795645 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 8586
W20260131 03:14:05.799868 8568 connection.cc:537] client connection to 127.0.0.1:41325 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41325): Connection reset by peer (error 104)
2026-01-31T03:14:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:14:05 dist-test-slave-5nnp krb5kdc[8685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:14:05 dist-test-slave-5nnp krb5kdc[8685](info): set up 2 sockets
Jan 31 03:14:05 dist-test-slave-5nnp krb5kdc[8685](info): commencing operation
krb5kdc: starting...
W20260131 03:14:07.795974 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.982s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:14:07 dist-test-slave-5nnp krb5kdc[8685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:14:07Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:14:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:14:07.934499 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44831
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:37577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:14:08.015658 8701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:14:08.015865 8701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:14:08.015902 8701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:14:08.017347 8701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:14:08.017422 8701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:14:08.017447 8701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:14:08.017486 8701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:14:08.017510 8701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:14:08.019117 8701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:14:08.019389 8701 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:14:08.019595 8701 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:14:08.022142 8706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:08.022149 8709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:08.022351 8701 server_base.cc:1060] running on GCE node
W20260131 03:14:08.022238 8707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:08.022693 8701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:14:08.022964 8701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:14:08.024122 8701 hybrid_clock.cc:648] HybridClock initialized: now 1769829248024098 us; error 45 us; skew 500 ppm
Jan 31 03:14:08 dist-test-slave-5nnp krb5kdc[8685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:14:08.027280 8701 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260131 03:14:08.028079 8701 webserver.cc:492] Webserver started at http://127.0.0.1:33725/ using document root <none> and password file <none>
I20260131 03:14:08.028312 8701 fs_manager.cc:362] Metadata directory not provided
I20260131 03:14:08.028364 8701 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:14:08.028508 8701 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:14:08.029873 8701 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "0d1e9fcb29f8466e87fdbe42c1293ad7"
format_stamp: "Formatted at 2026-01-31 03:14:08 on dist-test-slave-5nnp"
server_key: "76622f7dbe209af059c8e524799b4539"
server_key_iv: "9f0443b5193f7593f826c974ad1b3ceb"
server_key_version: "encryptionkey@0"
I20260131 03:14:08.030244 8701 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "0d1e9fcb29f8466e87fdbe42c1293ad7"
format_stamp: "Formatted at 2026-01-31 03:14:08 on dist-test-slave-5nnp"
server_key: "76622f7dbe209af059c8e524799b4539"
server_key_iv: "9f0443b5193f7593f826c974ad1b3ceb"
server_key_version: "encryptionkey@0"
I20260131 03:14:08.031940 8701 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260131 03:14:08.032923 8716 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:14:08.033171 8701 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:14:08.033265 8701 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "0d1e9fcb29f8466e87fdbe42c1293ad7"
format_stamp: "Formatted at 2026-01-31 03:14:08 on dist-test-slave-5nnp"
server_key: "76622f7dbe209af059c8e524799b4539"
server_key_iv: "9f0443b5193f7593f826c974ad1b3ceb"
server_key_version: "encryptionkey@0"
I20260131 03:14:08.033335 8701 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:14:08.037529 8701 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:14:08.039522 8701 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:14:08.039619 8701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:14:08.043586 8701 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44831
I20260131 03:14:08.043660 8768 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:44831 every 8 connection(s)
I20260131 03:14:08.043958 8701 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:14:08.044560 8769 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:14:08.046938 8769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d1e9fcb29f8466e87fdbe42c1293ad7: Bootstrap starting.
I20260131 03:14:08.047554 8769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d1e9fcb29f8466e87fdbe42c1293ad7: Neither blocks nor log segments found. Creating new log.
I20260131 03:14:08.047798 8769 log.cc:826] T 00000000000000000000000000000000 P 0d1e9fcb29f8466e87fdbe42c1293ad7: Log is configured to *not* fsync() on all Append() calls
I20260131 03:14:08.048455 8769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d1e9fcb29f8466e87fdbe42c1293ad7: No bootstrap required, opened a new log
I20260131 03:14:08.049362 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 8701
I20260131 03:14:08.049495 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:14:08.049670 19792 external_mini_cluster.cc:1468] Setting key 5c480557940ab0da73e2cf0e53b16f13
I20260131 03:14:08.049955 8769 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0d1e9fcb29f8466e87fdbe42c1293ad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1e9fcb29f8466e87fdbe42c1293ad7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } }
I20260131 03:14:08.050089 8769 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0d1e9fcb29f8466e87fdbe42c1293ad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:14:08.050119 8769 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0d1e9fcb29f8466e87fdbe42c1293ad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d1e9fcb29f8466e87fdbe42c1293ad7, State: Initialized, Role: FOLLOWER
I20260131 03:14:08.050227 8769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d1e9fcb29f8466e87fdbe42c1293ad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1e9fcb29f8466e87fdbe42c1293ad7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } }
I20260131 03:14:08.050307 8769 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0d1e9fcb29f8466e87fdbe42c1293ad7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:14:08.050331 8769 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0d1e9fcb29f8466e87fdbe42c1293ad7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:14:08.050380 8769 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0d1e9fcb29f8466e87fdbe42c1293ad7 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:14:08.050887 8769 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0d1e9fcb29f8466e87fdbe42c1293ad7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1e9fcb29f8466e87fdbe42c1293ad7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } }
I20260131 03:14:08.051026 8769 leader_election.cc:304] T 00000000000000000000000000000000 P 0d1e9fcb29f8466e87fdbe42c1293ad7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d1e9fcb29f8466e87fdbe42c1293ad7; no voters:
I20260131 03:14:08.051172 8769 leader_election.cc:290] T 00000000000000000000000000000000 P 0d1e9fcb29f8466e87fdbe42c1293ad7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:14:08.051254 8774 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0d1e9fcb29f8466e87fdbe42c1293ad7 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:14:08.051412 8774 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0d1e9fcb29f8466e87fdbe42c1293ad7 [term 1 LEADER]: Becoming Leader. State: Replica: 0d1e9fcb29f8466e87fdbe42c1293ad7, State: Running, Role: LEADER
I20260131 03:14:08.051482 8769 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0d1e9fcb29f8466e87fdbe42c1293ad7 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:14:08.051504 8774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d1e9fcb29f8466e87fdbe42c1293ad7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1e9fcb29f8466e87fdbe42c1293ad7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } }
I20260131 03:14:08.051927 8773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:08.050160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:45461 (local address 127.0.0.1:44831)
0131 03:14:08.050328 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:08.050343 (+ 15us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:08.050368 (+ 25us) server_negotiation.cc:408] Connection header received
0131 03:14:08.050555 (+ 187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:08.050560 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:08.050583 (+ 23us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:08.050695 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:08.051433 (+ 738us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:08.051436 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:08.051436 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:14:08.051440 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:08.051463 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:14:08.051480 (+ 17us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:08.051699 (+ 219us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:08.051733 (+ 34us) server_negotiation.cc:300] Negotiation successful
0131 03:14:08.051768 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
I20260131 03:14:08.051891 8776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d1e9fcb29f8466e87fdbe42c1293ad7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d1e9fcb29f8466e87fdbe42c1293ad7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1e9fcb29f8466e87fdbe42c1293ad7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } } }
I20260131 03:14:08.051978 8776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d1e9fcb29f8466e87fdbe42c1293ad7 [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:08.051880 8774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d1e9fcb29f8466e87fdbe42c1293ad7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d1e9fcb29f8466e87fdbe42c1293ad7. Latest consensus state: current_term: 1 leader_uuid: "0d1e9fcb29f8466e87fdbe42c1293ad7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d1e9fcb29f8466e87fdbe42c1293ad7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } } }
I20260131 03:14:08.052161 8774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d1e9fcb29f8466e87fdbe42c1293ad7 [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:08.052175 8779 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:14:08.052626 8779 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:14:08.053315 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:14:08.055215 8779 catalog_manager.cc:1357] Generated new cluster ID: 9dabddc7f8ce49c695db961f6367c7d0
I20260131 03:14:08.055271 8779 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:14:08.056437 8773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:08.054918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:43441 (local address 127.0.0.1:44831)
0131 03:14:08.054974 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:08.054976 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:08.055431 (+ 455us) server_negotiation.cc:408] Connection header received
0131 03:14:08.055549 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:08.055550 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:08.055555 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:08.055625 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:08.056201 (+ 576us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:08.056202 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:08.056203 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:14:08.056207 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:08.056222 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:14:08.056226 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:08.056283 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:08.056336 (+ 53us) server_negotiation.cc:300] Negotiation successful
0131 03:14:08.056352 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260131 03:14:08.062047 8779 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:14:08.062214 8779 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:14:08.066969 8779 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 0d1e9fcb29f8466e87fdbe42c1293ad7: Generated new TSK 0
I20260131 03:14:08.067129 8779 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:14:08.071000 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 8701
2026-01-31T03:14:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2275 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:14:08 dist-test-slave-5nnp krb5kdc[8800](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:14:08 dist-test-slave-5nnp krb5kdc[8800](info): set up 2 sockets
Jan 31 03:14:08 dist-test-slave-5nnp krb5kdc[8800](info): commencing operation
krb5kdc: starting...
W20260131 03:14:10.072029 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.983s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:14:10 dist-test-slave-5nnp krb5kdc[8800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:14:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:14:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:14:10.210527 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35585
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:37183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:14:10.289932 8816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:14:10.290184 8816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:14:10.290225 8816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:14:10.291780 8816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:14:10.291865 8816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:14:10.291889 8816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:14:10.291910 8816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:14:10.291941 8816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:14:10.293606 8816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35585
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:14:10.293897 8816 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:14:10.294101 8816 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:14:10.296711 8821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:10.296702 8824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:10.296701 8822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:10.297013 8816 server_base.cc:1060] running on GCE node
I20260131 03:14:10.297236 8816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:14:10.297515 8816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:14:10.298717 8816 hybrid_clock.cc:648] HybridClock initialized: now 1769829250298704 us; error 27 us; skew 500 ppm
Jan 31 03:14:10 dist-test-slave-5nnp krb5kdc[8800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:14:10.301146 8816 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260131 03:14:10.301750 8816 webserver.cc:492] Webserver started at http://127.0.0.1:37427/ using document root <none> and password file <none>
I20260131 03:14:10.301963 8816 fs_manager.cc:362] Metadata directory not provided
I20260131 03:14:10.302016 8816 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:14:10.302152 8816 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:14:10.303148 8816 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "fd4fdcd91554446cb470f5fcdeb98be7"
format_stamp: "Formatted at 2026-01-31 03:14:10 on dist-test-slave-5nnp"
server_key: "3ddf4922217b4d55be926c471b7a81ce"
server_key_iv: "728c5616e394f276e80bd4cce9bfdcf2"
server_key_version: "encryptionkey@0"
I20260131 03:14:10.303568 8816 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "fd4fdcd91554446cb470f5fcdeb98be7"
format_stamp: "Formatted at 2026-01-31 03:14:10 on dist-test-slave-5nnp"
server_key: "3ddf4922217b4d55be926c471b7a81ce"
server_key_iv: "728c5616e394f276e80bd4cce9bfdcf2"
server_key_version: "encryptionkey@0"
I20260131 03:14:10.304849 8816 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:14:10.305579 8831 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:14:10.305819 8816 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:14:10.305909 8816 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "fd4fdcd91554446cb470f5fcdeb98be7"
format_stamp: "Formatted at 2026-01-31 03:14:10 on dist-test-slave-5nnp"
server_key: "3ddf4922217b4d55be926c471b7a81ce"
server_key_iv: "728c5616e394f276e80bd4cce9bfdcf2"
server_key_version: "encryptionkey@0"
I20260131 03:14:10.305986 8816 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:14:10.335218 8816 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:14:10.337159 8816 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:14:10.337323 8816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:14:10.341804 8816 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35585
I20260131 03:14:10.341852 8883 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:35585 every 8 connection(s)
I20260131 03:14:10.342187 8816 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:14:10.342799 8884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:14:10.345355 8884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd4fdcd91554446cb470f5fcdeb98be7: Bootstrap starting.
I20260131 03:14:10.345400 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 8816
I20260131 03:14:10.345548 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:14:10.345757 19792 external_mini_cluster.cc:1468] Setting key 17f563080b51677f94b8466d3150abe4
I20260131 03:14:10.346086 8884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd4fdcd91554446cb470f5fcdeb98be7: Neither blocks nor log segments found. Creating new log.
I20260131 03:14:10.346380 8884 log.cc:826] T 00000000000000000000000000000000 P fd4fdcd91554446cb470f5fcdeb98be7: Log is configured to *not* fsync() on all Append() calls
I20260131 03:14:10.347246 8884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd4fdcd91554446cb470f5fcdeb98be7: No bootstrap required, opened a new log
I20260131 03:14:10.348637 8884 raft_consensus.cc:359] T 00000000000000000000000000000000 P fd4fdcd91554446cb470f5fcdeb98be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd4fdcd91554446cb470f5fcdeb98be7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35585 } }
I20260131 03:14:10.348749 8884 raft_consensus.cc:385] T 00000000000000000000000000000000 P fd4fdcd91554446cb470f5fcdeb98be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:14:10.348788 8884 raft_consensus.cc:740] T 00000000000000000000000000000000 P fd4fdcd91554446cb470f5fcdeb98be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd4fdcd91554446cb470f5fcdeb98be7, State: Initialized, Role: FOLLOWER
I20260131 03:14:10.348896 8884 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd4fdcd91554446cb470f5fcdeb98be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd4fdcd91554446cb470f5fcdeb98be7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35585 } }
I20260131 03:14:10.348961 8884 raft_consensus.cc:399] T 00000000000000000000000000000000 P fd4fdcd91554446cb470f5fcdeb98be7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:14:10.348984 8884 raft_consensus.cc:493] T 00000000000000000000000000000000 P fd4fdcd91554446cb470f5fcdeb98be7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:14:10.349050 8884 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fd4fdcd91554446cb470f5fcdeb98be7 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:14:10.349747 8884 raft_consensus.cc:515] T 00000000000000000000000000000000 P fd4fdcd91554446cb470f5fcdeb98be7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd4fdcd91554446cb470f5fcdeb98be7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35585 } }
I20260131 03:14:10.349886 8884 leader_election.cc:304] T 00000000000000000000000000000000 P fd4fdcd91554446cb470f5fcdeb98be7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd4fdcd91554446cb470f5fcdeb98be7; no voters:
I20260131 03:14:10.350036 8884 leader_election.cc:290] T 00000000000000000000000000000000 P fd4fdcd91554446cb470f5fcdeb98be7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:14:10.350095 8889 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fd4fdcd91554446cb470f5fcdeb98be7 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:14:10.350251 8889 raft_consensus.cc:697] T 00000000000000000000000000000000 P fd4fdcd91554446cb470f5fcdeb98be7 [term 1 LEADER]: Becoming Leader. State: Replica: fd4fdcd91554446cb470f5fcdeb98be7, State: Running, Role: LEADER
I20260131 03:14:10.350324 8884 sys_catalog.cc:565] T 00000000000000000000000000000000 P fd4fdcd91554446cb470f5fcdeb98be7 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:14:10.350457 8889 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd4fdcd91554446cb470f5fcdeb98be7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd4fdcd91554446cb470f5fcdeb98be7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35585 } }
I20260131 03:14:10.350878 8890 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4fdcd91554446cb470f5fcdeb98be7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd4fdcd91554446cb470f5fcdeb98be7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd4fdcd91554446cb470f5fcdeb98be7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35585 } } }
I20260131 03:14:10.350993 8890 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4fdcd91554446cb470f5fcdeb98be7 [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:10.351269 8891 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd4fdcd91554446cb470f5fcdeb98be7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd4fdcd91554446cb470f5fcdeb98be7. Latest consensus state: current_term: 1 leader_uuid: "fd4fdcd91554446cb470f5fcdeb98be7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd4fdcd91554446cb470f5fcdeb98be7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35585 } } }
I20260131 03:14:10.351413 8891 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd4fdcd91554446cb470f5fcdeb98be7 [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:10.351549 8898 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:14:10.352027 8898 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:14:10.353425 8898 catalog_manager.cc:1357] Generated new cluster ID: 0c438d0e876b45e5982f1a1b1e33c802
I20260131 03:14:10.353475 8898 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Jan 31 03:14:10 dist-test-slave-5nnp krb5kdc[8800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260131 03:14:10.358114 8887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:10.346293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:45929 (local address 127.0.0.1:35585)
0131 03:14:10.346458 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:10.346469 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:10.346533 (+ 64us) server_negotiation.cc:408] Connection header received
0131 03:14:10.346999 (+ 466us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:10.347006 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:10.347057 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:10.347164 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:10.348176 (+ 1012us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:10.349116 (+ 940us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:10.349939 (+ 823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:10.350236 (+ 297us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:10.354818 (+ 4582us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:10.354845 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:10.354845 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:10.354871 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:10.356718 (+ 1847us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:10.357114 (+ 396us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:10.357119 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:10.357119 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:10.357182 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:10.357387 (+ 205us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:10.357388 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:10.357389 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:10.357581 (+ 192us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:10.357751 (+ 170us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:10.357879 (+ 128us) server_negotiation.cc:300] Negotiation successful
0131 03:14:10.357986 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260131 03:14:10.359125 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:14:10.369904 8887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:10.360771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:52647 (local address 127.0.0.1:35585)
0131 03:14:10.360814 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:10.360816 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:10.363599 (+ 2783us) server_negotiation.cc:408] Connection header received
0131 03:14:10.363713 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:10.363714 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:10.363732 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:10.363825 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:10.364565 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:10.365104 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:10.365731 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:10.366319 (+ 588us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:10.367038 (+ 719us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:10.367043 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:10.367043 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:10.367053 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:10.368731 (+ 1678us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:10.369114 (+ 383us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:10.369115 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:10.369116 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:10.369145 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:10.369337 (+ 192us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:10.369338 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:10.369338 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:10.369423 (+ 85us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:10.369539 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:10.369719 (+ 180us) server_negotiation.cc:300] Negotiation successful
0131 03:14:10.369791 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260131 03:14:10.370460 8898 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:14:10.371143 8898 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:14:10.375576 8898 catalog_manager.cc:6027] T 00000000000000000000000000000000 P fd4fdcd91554446cb470f5fcdeb98be7: Generated new TSK 0
I20260131 03:14:10.375773 8898 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:14:10.383524 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 8816
W20260131 03:14:10.388051 8798 connection.cc:537] client connection to 127.0.0.1:35585 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35585): Connection reset by peer (error 104)
2026-01-31T03:14:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:14:10 dist-test-slave-5nnp krb5kdc[8915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:14:10 dist-test-slave-5nnp krb5kdc[8915](info): set up 2 sockets
Jan 31 03:14:10 dist-test-slave-5nnp krb5kdc[8915](info): commencing operation
krb5kdc: starting...
W20260131 03:14:12.388015 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.986s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:14:12 dist-test-slave-5nnp krb5kdc[8915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:14:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:14:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:14:12.522495 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46481
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:35181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:14:12.599831 8931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:14:12.600047 8931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:14:12.600085 8931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:14:12.601500 8931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:14:12.601567 8931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:14:12.601591 8931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:14:12.601614 8931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:14:12.601650 8931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:14:12.603194 8931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46481
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:14:12.603447 8931 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:14:12.603636 8931 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:14:12.605973 8939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:12.606139 8937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:12.606276 8931 server_base.cc:1060] running on GCE node
W20260131 03:14:12.606346 8936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:12.606600 8931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:14:12.606804 8931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:14:12.607934 8931 hybrid_clock.cc:648] HybridClock initialized: now 1769829252607916 us; error 33 us; skew 500 ppm
Jan 31 03:14:12 dist-test-slave-5nnp krb5kdc[8915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:14:12.609997 8931 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260131 03:14:12.610492 8931 webserver.cc:492] Webserver started at http://127.0.0.1:41077/ using document root <none> and password file <none>
I20260131 03:14:12.610678 8931 fs_manager.cc:362] Metadata directory not provided
I20260131 03:14:12.610714 8931 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:14:12.610841 8931 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:14:12.611817 8931 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "3007a890f3634c8dabf372dc9d73a02a"
format_stamp: "Formatted at 2026-01-31 03:14:12 on dist-test-slave-5nnp"
server_key: "2784497ad41811ea8e688069e6cc5838"
server_key_iv: "8072c45751831262f9ff73ad11f91299"
server_key_version: "encryptionkey@0"
I20260131 03:14:12.612136 8931 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "3007a890f3634c8dabf372dc9d73a02a"
format_stamp: "Formatted at 2026-01-31 03:14:12 on dist-test-slave-5nnp"
server_key: "2784497ad41811ea8e688069e6cc5838"
server_key_iv: "8072c45751831262f9ff73ad11f91299"
server_key_version: "encryptionkey@0"
I20260131 03:14:12.613261 8931 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:14:12.613916 8946 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:14:12.614087 8931 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260131 03:14:12.614163 8931 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "3007a890f3634c8dabf372dc9d73a02a"
format_stamp: "Formatted at 2026-01-31 03:14:12 on dist-test-slave-5nnp"
server_key: "2784497ad41811ea8e688069e6cc5838"
server_key_iv: "8072c45751831262f9ff73ad11f91299"
server_key_version: "encryptionkey@0"
I20260131 03:14:12.614228 8931 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:14:12.657930 8931 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:14:12.659763 8931 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:14:12.659925 8931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:14:12.663892 8931 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46481
I20260131 03:14:12.663961 8998 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:46481 every 8 connection(s)
I20260131 03:14:12.664253 8931 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:14:12.664860 8999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:14:12.667076 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 8931
I20260131 03:14:12.667182 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:14:12.667359 19792 external_mini_cluster.cc:1468] Setting key 0dae6350fe323bc0a442aa43cce67212
I20260131 03:14:12.667351 8999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3007a890f3634c8dabf372dc9d73a02a: Bootstrap starting.
I20260131 03:14:12.668061 8999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3007a890f3634c8dabf372dc9d73a02a: Neither blocks nor log segments found. Creating new log.
I20260131 03:14:12.668314 8999 log.cc:826] T 00000000000000000000000000000000 P 3007a890f3634c8dabf372dc9d73a02a: Log is configured to *not* fsync() on all Append() calls
I20260131 03:14:12.669121 8999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3007a890f3634c8dabf372dc9d73a02a: No bootstrap required, opened a new log
I20260131 03:14:12.670339 8999 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3007a890f3634c8dabf372dc9d73a02a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3007a890f3634c8dabf372dc9d73a02a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } }
I20260131 03:14:12.670488 8999 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3007a890f3634c8dabf372dc9d73a02a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:14:12.670524 8999 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3007a890f3634c8dabf372dc9d73a02a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3007a890f3634c8dabf372dc9d73a02a, State: Initialized, Role: FOLLOWER
I20260131 03:14:12.670696 8999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3007a890f3634c8dabf372dc9d73a02a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3007a890f3634c8dabf372dc9d73a02a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } }
I20260131 03:14:12.670780 8999 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3007a890f3634c8dabf372dc9d73a02a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:14:12.670811 8999 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3007a890f3634c8dabf372dc9d73a02a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:14:12.670918 8999 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3007a890f3634c8dabf372dc9d73a02a [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:14:12.671665 8999 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3007a890f3634c8dabf372dc9d73a02a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3007a890f3634c8dabf372dc9d73a02a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } }
I20260131 03:14:12.671774 8999 leader_election.cc:304] T 00000000000000000000000000000000 P 3007a890f3634c8dabf372dc9d73a02a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3007a890f3634c8dabf372dc9d73a02a; no voters:
I20260131 03:14:12.671912 9002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:12.667885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:52703 (local address 127.0.0.1:46481)
0131 03:14:12.668043 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:12.668052 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:12.668071 (+ 19us) server_negotiation.cc:408] Connection header received
0131 03:14:12.668389 (+ 318us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:12.668407 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:12.668460 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:12.668598 (+ 138us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:12.669014 (+ 416us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:12.669762 (+ 748us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:12.670530 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:12.670800 (+ 270us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:12.671228 (+ 428us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:12.671235 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:12.671235 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:14:12.671437 (+ 202us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:12.671455 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:14:12.671461 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:12.671530 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:12.671693 (+ 163us) server_negotiation.cc:300] Negotiation successful
0131 03:14:12.671722 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260131 03:14:12.671962 8999 leader_election.cc:290] T 00000000000000000000000000000000 P 3007a890f3634c8dabf372dc9d73a02a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:14:12.671993 9004 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3007a890f3634c8dabf372dc9d73a02a [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:14:12.672089 9004 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3007a890f3634c8dabf372dc9d73a02a [term 1 LEADER]: Becoming Leader. State: Replica: 3007a890f3634c8dabf372dc9d73a02a, State: Running, Role: LEADER
I20260131 03:14:12.672188 8999 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3007a890f3634c8dabf372dc9d73a02a [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:14:12.672179 9004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3007a890f3634c8dabf372dc9d73a02a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3007a890f3634c8dabf372dc9d73a02a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } }
I20260131 03:14:12.672487 9005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3007a890f3634c8dabf372dc9d73a02a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3007a890f3634c8dabf372dc9d73a02a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3007a890f3634c8dabf372dc9d73a02a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } } }
I20260131 03:14:12.672602 9005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3007a890f3634c8dabf372dc9d73a02a [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:12.672474 9004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3007a890f3634c8dabf372dc9d73a02a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3007a890f3634c8dabf372dc9d73a02a. Latest consensus state: current_term: 1 leader_uuid: "3007a890f3634c8dabf372dc9d73a02a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3007a890f3634c8dabf372dc9d73a02a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } } }
I20260131 03:14:12.672752 9004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3007a890f3634c8dabf372dc9d73a02a [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:12.672870 9008 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:14:12.673166 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:14:12.673491 9008 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:14:12.674966 9008 catalog_manager.cc:1357] Generated new cluster ID: 22cbad00b159443e83debe3f618fc14e
I20260131 03:14:12.675010 9008 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:14:12.677385 9002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:12.674704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:44927 (local address 127.0.0.1:46481)
0131 03:14:12.674763 (+ 59us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:12.674765 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:12.674933 (+ 168us) server_negotiation.cc:408] Connection header received
0131 03:14:12.674956 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:12.674957 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:12.674965 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:12.675023 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:12.675284 (+ 261us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:12.675774 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:12.676342 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:12.676525 (+ 183us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:12.677089 (+ 564us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:12.677092 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:12.677092 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:14:12.677097 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:12.677112 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:14:12.677117 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:12.677169 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:12.677280 (+ 111us) server_negotiation.cc:300] Negotiation successful
0131 03:14:12.677290 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260131 03:14:12.689467 9008 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:14:12.690138 9008 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:14:12.695080 9008 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 3007a890f3634c8dabf372dc9d73a02a: Generated new TSK 0
I20260131 03:14:12.695237 9008 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:14:12.697466 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 8931
W20260131 03:14:12.701699 8913 connection.cc:537] client connection to 127.0.0.1:46481 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46481): Connection reset by peer (error 104)
2026-01-31T03:14:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:14:12 dist-test-slave-5nnp krb5kdc[9030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:14:12 dist-test-slave-5nnp krb5kdc[9030](info): set up 2 sockets
Jan 31 03:14:12 dist-test-slave-5nnp krb5kdc[9030](info): commencing operation
krb5kdc: starting...
W20260131 03:14:14.704066 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.988s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:14:14 dist-test-slave-5nnp krb5kdc[9030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:14:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:14:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:14:14.838801 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45121
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:38453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:14:14.920152 9046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:14:14.920331 9046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:14:14.920353 9046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:14:14.921893 9046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:14:14.921941 9046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:14:14.921955 9046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:14:14.922009 9046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:14:14.922032 9046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:14:14.923710 9046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45121
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:14:14.923960 9046 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:14:14.924166 9046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:14:14.926683 9052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:14.926687 9054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:14.926767 9051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:14.926859 9046 server_base.cc:1060] running on GCE node
I20260131 03:14:14.927037 9046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:14:14.927232 9046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:14:14.928383 9046 hybrid_clock.cc:648] HybridClock initialized: now 1769829254928372 us; error 33 us; skew 500 ppm
Jan 31 03:14:14 dist-test-slave-5nnp krb5kdc[9030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:14:14.930531 9046 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260131 03:14:14.931077 9046 webserver.cc:492] Webserver started at http://127.0.0.1:40705/ using document root <none> and password file <none>
I20260131 03:14:14.931263 9046 fs_manager.cc:362] Metadata directory not provided
I20260131 03:14:14.931313 9046 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:14:14.931456 9046 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:14:14.932389 9046 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "1a25ff0b2d22437cb5139ae28e2ad90a"
format_stamp: "Formatted at 2026-01-31 03:14:14 on dist-test-slave-5nnp"
server_key: "2612254e1aa5bb8939aa1cad41fa7d5d"
server_key_iv: "9dd53de8ff610112677682231c448acf"
server_key_version: "encryptionkey@0"
I20260131 03:14:14.932731 9046 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "1a25ff0b2d22437cb5139ae28e2ad90a"
format_stamp: "Formatted at 2026-01-31 03:14:14 on dist-test-slave-5nnp"
server_key: "2612254e1aa5bb8939aa1cad41fa7d5d"
server_key_iv: "9dd53de8ff610112677682231c448acf"
server_key_version: "encryptionkey@0"
I20260131 03:14:14.933964 9046 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:14:14.934590 9061 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:14:14.934782 9046 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260131 03:14:14.934867 9046 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "1a25ff0b2d22437cb5139ae28e2ad90a"
format_stamp: "Formatted at 2026-01-31 03:14:14 on dist-test-slave-5nnp"
server_key: "2612254e1aa5bb8939aa1cad41fa7d5d"
server_key_iv: "9dd53de8ff610112677682231c448acf"
server_key_version: "encryptionkey@0"
I20260131 03:14:14.934938 9046 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:14:14.938186 9046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:14:14.940088 9046 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:14:14.940220 9046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:14:14.944152 9046 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45121
I20260131 03:14:14.944181 9113 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:45121 every 8 connection(s)
I20260131 03:14:14.944476 9046 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:14:14.945062 9114 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:14:14.947507 9114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a25ff0b2d22437cb5139ae28e2ad90a: Bootstrap starting.
I20260131 03:14:14.948040 9114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a25ff0b2d22437cb5139ae28e2ad90a: Neither blocks nor log segments found. Creating new log.
I20260131 03:14:14.948236 9114 log.cc:826] T 00000000000000000000000000000000 P 1a25ff0b2d22437cb5139ae28e2ad90a: Log is configured to *not* fsync() on all Append() calls
I20260131 03:14:14.948911 9114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a25ff0b2d22437cb5139ae28e2ad90a: No bootstrap required, opened a new log
I20260131 03:14:14.950052 9114 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1a25ff0b2d22437cb5139ae28e2ad90a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a25ff0b2d22437cb5139ae28e2ad90a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45121 } }
I20260131 03:14:14.950168 9114 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1a25ff0b2d22437cb5139ae28e2ad90a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:14:14.950189 9114 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1a25ff0b2d22437cb5139ae28e2ad90a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a25ff0b2d22437cb5139ae28e2ad90a, State: Initialized, Role: FOLLOWER
I20260131 03:14:14.950313 9114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a25ff0b2d22437cb5139ae28e2ad90a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a25ff0b2d22437cb5139ae28e2ad90a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45121 } }
I20260131 03:14:14.950392 9114 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1a25ff0b2d22437cb5139ae28e2ad90a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:14:14.950414 9114 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1a25ff0b2d22437cb5139ae28e2ad90a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:14:14.950446 9114 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1a25ff0b2d22437cb5139ae28e2ad90a [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:14:14.950937 9114 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1a25ff0b2d22437cb5139ae28e2ad90a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a25ff0b2d22437cb5139ae28e2ad90a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45121 } }
I20260131 03:14:14.951021 9114 leader_election.cc:304] T 00000000000000000000000000000000 P 1a25ff0b2d22437cb5139ae28e2ad90a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a25ff0b2d22437cb5139ae28e2ad90a; no voters:
I20260131 03:14:14.951131 9114 leader_election.cc:290] T 00000000000000000000000000000000 P 1a25ff0b2d22437cb5139ae28e2ad90a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:14:14.951200 9117 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1a25ff0b2d22437cb5139ae28e2ad90a [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:14:14.951341 9117 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1a25ff0b2d22437cb5139ae28e2ad90a [term 1 LEADER]: Becoming Leader. State: Replica: 1a25ff0b2d22437cb5139ae28e2ad90a, State: Running, Role: LEADER
I20260131 03:14:14.951407 9114 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1a25ff0b2d22437cb5139ae28e2ad90a [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:14:14.951479 9117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a25ff0b2d22437cb5139ae28e2ad90a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a25ff0b2d22437cb5139ae28e2ad90a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45121 } }
I20260131 03:14:14.951838 9119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a25ff0b2d22437cb5139ae28e2ad90a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a25ff0b2d22437cb5139ae28e2ad90a. Latest consensus state: current_term: 1 leader_uuid: "1a25ff0b2d22437cb5139ae28e2ad90a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a25ff0b2d22437cb5139ae28e2ad90a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45121 } } }
I20260131 03:14:14.951923 9119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a25ff0b2d22437cb5139ae28e2ad90a [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:14.951854 9118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a25ff0b2d22437cb5139ae28e2ad90a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a25ff0b2d22437cb5139ae28e2ad90a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a25ff0b2d22437cb5139ae28e2ad90a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45121 } } }
I20260131 03:14:14.952041 9118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a25ff0b2d22437cb5139ae28e2ad90a [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:14.952159 9123 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:14:14.952816 9123 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:14:14.952965 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 9046
I20260131 03:14:14.953084 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:14:14.953248 19792 external_mini_cluster.cc:1468] Setting key 0c380f64308f91a3138036876bd05777
I20260131 03:14:14.954559 9123 catalog_manager.cc:1357] Generated new cluster ID: 84b9b4a31e4c489bb28596da336e0572
I20260131 03:14:14.954612 9123 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:14:14.955494 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:14.953929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:54549 (local address 127.0.0.1:45121)
0131 03:14:14.954079 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:14.954090 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:14.954118 (+ 28us) server_negotiation.cc:408] Connection header received
0131 03:14:14.954237 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:14.954241 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:14.954263 (+ 22us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:14.954392 (+ 129us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:14.955086 (+ 694us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:14.955089 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:14.955089 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:14:14.955094 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:14.955121 (+ 27us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:14:14.955127 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:14.955233 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:14.955268 (+ 35us) server_negotiation.cc:300] Negotiation successful
0131 03:14:14.955310 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":86,"threads_started":1}
I20260131 03:14:14.956338 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:14:14.960546 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:14.957653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:51229 (local address 127.0.0.1:45121)
0131 03:14:14.957691 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:14.957693 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:14.959557 (+ 1864us) server_negotiation.cc:408] Connection header received
0131 03:14:14.959635 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:14.959636 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:14.959641 (+ 5us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:14.959693 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:14.960283 (+ 590us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:14.960284 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:14.960284 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:14:14.960288 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:14.960316 (+ 28us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:14:14.960321 (+ 5us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:14.960382 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:14.960441 (+ 59us) server_negotiation.cc:300] Negotiation successful
0131 03:14:14.960458 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260131 03:14:14.970137 9123 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:14:14.970307 9123 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:14:14.981242 9123 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 1a25ff0b2d22437cb5139ae28e2ad90a: Generated new TSK 0
I20260131 03:14:14.981459 9123 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:14:14.989207 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 9046
2026-01-31T03:14:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:14:15 dist-test-slave-5nnp krb5kdc[9145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:14:15 dist-test-slave-5nnp krb5kdc[9145](info): set up 2 sockets
Jan 31 03:14:15 dist-test-slave-5nnp krb5kdc[9145](info): commencing operation
krb5kdc: starting...
W20260131 03:14:16.996250 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.989s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:14:17 dist-test-slave-5nnp krb5kdc[9145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:14:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:14:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:14:17.138001 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:39755
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:35217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:39755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:14:17.216732 9161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:14:17.216938 9161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:14:17.216976 9161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:14:17.218403 9161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:14:17.218469 9161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:14:17.218492 9161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:14:17.218515 9161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:14:17.218551 9161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:14:17.220134 9161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:35217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:39755
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:39755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:14:17.220374 9161 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:14:17.220556 9161 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:14:17.222841 9169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:17.222853 9166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:17.222856 9167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:17.223133 9161 server_base.cc:1060] running on GCE node
I20260131 03:14:17.223343 9161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:14:17.223620 9161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:14:17.224764 9161 hybrid_clock.cc:648] HybridClock initialized: now 1769829257224746 us; error 28 us; skew 500 ppm
Jan 31 03:14:17 dist-test-slave-5nnp krb5kdc[9145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829257, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:14:17.227027 9161 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:14:17.227612 9161 webserver.cc:492] Webserver started at http://127.19.84.62:33337/ using document root <none> and password file <none>
I20260131 03:14:17.227844 9161 fs_manager.cc:362] Metadata directory not provided
I20260131 03:14:17.227888 9161 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:14:17.228001 9161 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:14:17.228951 9161 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "e72c354b95f44afb8632ac2250460faa"
format_stamp: "Formatted at 2026-01-31 03:14:17 on dist-test-slave-5nnp"
server_key: "a3c9f2bbc8351796dd18ee9ce6f0de52"
server_key_iv: "8eb680e2feb060a3f19d6d3be6d19e55"
server_key_version: "encryptionkey@0"
I20260131 03:14:17.229259 9161 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "e72c354b95f44afb8632ac2250460faa"
format_stamp: "Formatted at 2026-01-31 03:14:17 on dist-test-slave-5nnp"
server_key: "a3c9f2bbc8351796dd18ee9ce6f0de52"
server_key_iv: "8eb680e2feb060a3f19d6d3be6d19e55"
server_key_version: "encryptionkey@0"
I20260131 03:14:17.230486 9161 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:14:17.231240 9176 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:14:17.231465 9161 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:14:17.231547 9161 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "e72c354b95f44afb8632ac2250460faa"
format_stamp: "Formatted at 2026-01-31 03:14:17 on dist-test-slave-5nnp"
server_key: "a3c9f2bbc8351796dd18ee9ce6f0de52"
server_key_iv: "8eb680e2feb060a3f19d6d3be6d19e55"
server_key_version: "encryptionkey@0"
I20260131 03:14:17.231619 9161 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:14:17.248848 9161 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:14:17.250555 9161 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:14:17.250710 9161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:14:17.254976 9161 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:39755
I20260131 03:14:17.255025 9228 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:39755 every 8 connection(s)
I20260131 03:14:17.255353 9161 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:14:17.256009 9229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:14:17.258376 9229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e72c354b95f44afb8632ac2250460faa: Bootstrap starting.
I20260131 03:14:17.258910 9229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e72c354b95f44afb8632ac2250460faa: Neither blocks nor log segments found. Creating new log.
I20260131 03:14:17.259107 9229 log.cc:826] T 00000000000000000000000000000000 P e72c354b95f44afb8632ac2250460faa: Log is configured to *not* fsync() on all Append() calls
I20260131 03:14:17.259797 9229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e72c354b95f44afb8632ac2250460faa: No bootstrap required, opened a new log
I20260131 03:14:17.260948 9229 raft_consensus.cc:359] T 00000000000000000000000000000000 P e72c354b95f44afb8632ac2250460faa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72c354b95f44afb8632ac2250460faa" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 39755 } }
I20260131 03:14:17.261059 9229 raft_consensus.cc:385] T 00000000000000000000000000000000 P e72c354b95f44afb8632ac2250460faa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:14:17.261080 9229 raft_consensus.cc:740] T 00000000000000000000000000000000 P e72c354b95f44afb8632ac2250460faa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72c354b95f44afb8632ac2250460faa, State: Initialized, Role: FOLLOWER
I20260131 03:14:17.261190 9229 consensus_queue.cc:260] T 00000000000000000000000000000000 P e72c354b95f44afb8632ac2250460faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72c354b95f44afb8632ac2250460faa" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 39755 } }
I20260131 03:14:17.261264 9229 raft_consensus.cc:399] T 00000000000000000000000000000000 P e72c354b95f44afb8632ac2250460faa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:14:17.261287 9229 raft_consensus.cc:493] T 00000000000000000000000000000000 P e72c354b95f44afb8632ac2250460faa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:14:17.261312 9229 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e72c354b95f44afb8632ac2250460faa [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:14:17.261782 9229 raft_consensus.cc:515] T 00000000000000000000000000000000 P e72c354b95f44afb8632ac2250460faa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72c354b95f44afb8632ac2250460faa" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 39755 } }
I20260131 03:14:17.261870 9229 leader_election.cc:304] T 00000000000000000000000000000000 P e72c354b95f44afb8632ac2250460faa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e72c354b95f44afb8632ac2250460faa; no voters:
I20260131 03:14:17.262013 9229 leader_election.cc:290] T 00000000000000000000000000000000 P e72c354b95f44afb8632ac2250460faa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:14:17.262048 9232 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e72c354b95f44afb8632ac2250460faa [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:14:17.262153 9232 raft_consensus.cc:697] T 00000000000000000000000000000000 P e72c354b95f44afb8632ac2250460faa [term 1 LEADER]: Becoming Leader. State: Replica: e72c354b95f44afb8632ac2250460faa, State: Running, Role: LEADER
I20260131 03:14:17.262246 9229 sys_catalog.cc:565] T 00000000000000000000000000000000 P e72c354b95f44afb8632ac2250460faa [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:14:17.262295 9232 consensus_queue.cc:237] T 00000000000000000000000000000000 P e72c354b95f44afb8632ac2250460faa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72c354b95f44afb8632ac2250460faa" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 39755 } }
I20260131 03:14:17.262755 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 9161
I20260131 03:14:17.262880 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:14:17.263069 19792 external_mini_cluster.cc:1468] Setting key 89e3d891e21f3dbcf732c4b6ccdaf478
I20260131 03:14:17.263058 9233 sys_catalog.cc:455] T 00000000000000000000000000000000 P e72c354b95f44afb8632ac2250460faa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e72c354b95f44afb8632ac2250460faa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72c354b95f44afb8632ac2250460faa" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 39755 } } }
I20260131 03:14:17.263152 9233 sys_catalog.cc:458] T 00000000000000000000000000000000 P e72c354b95f44afb8632ac2250460faa [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:17.263437 9234 sys_catalog.cc:455] T 00000000000000000000000000000000 P e72c354b95f44afb8632ac2250460faa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e72c354b95f44afb8632ac2250460faa. Latest consensus state: current_term: 1 leader_uuid: "e72c354b95f44afb8632ac2250460faa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72c354b95f44afb8632ac2250460faa" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 39755 } } }
I20260131 03:14:17.263515 9234 sys_catalog.cc:458] T 00000000000000000000000000000000 P e72c354b95f44afb8632ac2250460faa [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:17.263495 9242 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:14:17.264251 9242 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:14:17.265545 9242 catalog_manager.cc:1357] Generated new cluster ID: 160697965878491d86cb897266b47efe
I20260131 03:14:17.265599 9242 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Jan 31 03:14:17 dist-test-slave-5nnp krb5kdc[9145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:14:17.272912 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:17.263696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53339 (local address 127.19.84.62:39755)
0131 03:14:17.263832 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:17.263844 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:17.263957 (+ 113us) server_negotiation.cc:408] Connection header received
0131 03:14:17.264103 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:17.264107 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:17.264143 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:17.264259 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:17.265136 (+ 877us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:17.266083 (+ 947us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:17.266649 (+ 566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:17.266923 (+ 274us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:17.269360 (+ 2437us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:17.269381 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:17.269382 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:17.269401 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:17.271437 (+ 2036us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:17.271839 (+ 402us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:17.271842 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:17.271842 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:17.271911 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:17.272107 (+ 196us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:17.272108 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:17.272108 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:17.272369 (+ 261us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:17.272517 (+ 148us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:17.272628 (+ 111us) server_negotiation.cc:300] Negotiation successful
0131 03:14:17.272734 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":72,"threads_started":1}
I20260131 03:14:17.273852 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:14:17.280442 9242 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:14:17.280988 9242 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:14:17.282392 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:17.275222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38699 (local address 127.19.84.62:39755)
0131 03:14:17.275263 (+ 41us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:17.275265 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:17.275463 (+ 198us) server_negotiation.cc:408] Connection header received
0131 03:14:17.275580 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:17.275581 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:17.275592 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:17.275676 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:17.276406 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:17.277440 (+ 1034us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:17.278114 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:17.278300 (+ 186us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:17.279534 (+ 1234us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:17.279538 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:17.279538 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:17.279546 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:17.281156 (+ 1610us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:17.281539 (+ 383us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:17.281541 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:17.281541 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:17.281566 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:17.281765 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:17.281766 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:17.281767 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:17.281879 (+ 112us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:17.281989 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:17.282120 (+ 131us) server_negotiation.cc:300] Negotiation successful
0131 03:14:17.282186 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260131 03:14:17.289379 9242 catalog_manager.cc:6027] T 00000000000000000000000000000000 P e72c354b95f44afb8632ac2250460faa: Generated new TSK 0
I20260131 03:14:17.289553 9242 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:14:17.293546 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 9161
W20260131 03:14:17.298218 9143 connection.cc:537] client connection to 127.19.84.62:39755 recv error: Network error: failed to read from TLS socket (remote: 127.19.84.62:39755): Connection reset by peer (error 104)
2026-01-31T03:14:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:14:17 dist-test-slave-5nnp krb5kdc[9261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:14:17 dist-test-slave-5nnp krb5kdc[9261](info): set up 2 sockets
Jan 31 03:14:17 dist-test-slave-5nnp krb5kdc[9261](info): commencing operation
krb5kdc: starting...
W20260131 03:14:19.295902 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.984s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:14:19 dist-test-slave-5nnp krb5kdc[9261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:14:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:14:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:14:19.425770 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:45577
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:45125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:45577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:14:19.504086 9277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:14:19.504264 9277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:14:19.504288 9277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:14:19.505771 9277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:14:19.505813 9277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:14:19.505828 9277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:14:19.505841 9277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:14:19.505852 9277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:14:19.507534 9277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:45125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:45577
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:45577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:14:19.507773 9277 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:14:19.507965 9277 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:14:19.510326 9283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:19.510361 9285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:19.510330 9282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:19.510524 9277 server_base.cc:1060] running on GCE node
I20260131 03:14:19.510767 9277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:14:19.511001 9277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:14:19.512148 9277 hybrid_clock.cc:648] HybridClock initialized: now 1769829259512144 us; error 35 us; skew 500 ppm
Jan 31 03:14:19 dist-test-slave-5nnp krb5kdc[9261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829259, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:14:19.514376 9277 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:14:19.514894 9277 webserver.cc:492] Webserver started at http://127.19.84.62:37169/ using document root <none> and password file <none>
I20260131 03:14:19.515090 9277 fs_manager.cc:362] Metadata directory not provided
I20260131 03:14:19.515141 9277 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:14:19.515264 9277 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:14:19.516294 9277 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "cb22b4d53b0c43f3a351d59e80cffcdb"
format_stamp: "Formatted at 2026-01-31 03:14:19 on dist-test-slave-5nnp"
server_key: "5bce458b83b66e371768b68ca7a569ae"
server_key_iv: "d2335d566252a515932263274475ccb3"
server_key_version: "encryptionkey@0"
I20260131 03:14:19.516607 9277 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "cb22b4d53b0c43f3a351d59e80cffcdb"
format_stamp: "Formatted at 2026-01-31 03:14:19 on dist-test-slave-5nnp"
server_key: "5bce458b83b66e371768b68ca7a569ae"
server_key_iv: "d2335d566252a515932263274475ccb3"
server_key_version: "encryptionkey@0"
I20260131 03:14:19.517830 9277 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260131 03:14:19.518487 9292 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:14:19.518682 9277 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260131 03:14:19.518772 9277 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "cb22b4d53b0c43f3a351d59e80cffcdb"
format_stamp: "Formatted at 2026-01-31 03:14:19 on dist-test-slave-5nnp"
server_key: "5bce458b83b66e371768b68ca7a569ae"
server_key_iv: "d2335d566252a515932263274475ccb3"
server_key_version: "encryptionkey@0"
I20260131 03:14:19.518842 9277 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:14:19.537570 9277 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:14:19.542555 9277 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:14:19.542722 9277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:14:19.546803 9277 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:45577
I20260131 03:14:19.546880 9344 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:45577 every 8 connection(s)
I20260131 03:14:19.547183 9277 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:14:19.547816 9345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:14:19.549998 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 9277
I20260131 03:14:19.550119 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:14:19.550292 19792 external_mini_cluster.cc:1468] Setting key 71e46fa1a99c441d3d429ca68d8f4384
I20260131 03:14:19.550552 9345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb22b4d53b0c43f3a351d59e80cffcdb: Bootstrap starting.
I20260131 03:14:19.551203 9345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb22b4d53b0c43f3a351d59e80cffcdb: Neither blocks nor log segments found. Creating new log.
I20260131 03:14:19.551520 9345 log.cc:826] T 00000000000000000000000000000000 P cb22b4d53b0c43f3a351d59e80cffcdb: Log is configured to *not* fsync() on all Append() calls
I20260131 03:14:19.552215 9345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb22b4d53b0c43f3a351d59e80cffcdb: No bootstrap required, opened a new log
I20260131 03:14:19.553261 9345 raft_consensus.cc:359] T 00000000000000000000000000000000 P cb22b4d53b0c43f3a351d59e80cffcdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb22b4d53b0c43f3a351d59e80cffcdb" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 45577 } }
I20260131 03:14:19.553400 9345 raft_consensus.cc:385] T 00000000000000000000000000000000 P cb22b4d53b0c43f3a351d59e80cffcdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:14:19.553459 9345 raft_consensus.cc:740] T 00000000000000000000000000000000 P cb22b4d53b0c43f3a351d59e80cffcdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb22b4d53b0c43f3a351d59e80cffcdb, State: Initialized, Role: FOLLOWER
I20260131 03:14:19.553584 9345 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb22b4d53b0c43f3a351d59e80cffcdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb22b4d53b0c43f3a351d59e80cffcdb" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 45577 } }
I20260131 03:14:19.553670 9345 raft_consensus.cc:399] T 00000000000000000000000000000000 P cb22b4d53b0c43f3a351d59e80cffcdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:14:19.553710 9345 raft_consensus.cc:493] T 00000000000000000000000000000000 P cb22b4d53b0c43f3a351d59e80cffcdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:14:19.553779 9345 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cb22b4d53b0c43f3a351d59e80cffcdb [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:14:19.554365 9345 raft_consensus.cc:515] T 00000000000000000000000000000000 P cb22b4d53b0c43f3a351d59e80cffcdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb22b4d53b0c43f3a351d59e80cffcdb" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 45577 } }
I20260131 03:14:19.554488 9345 leader_election.cc:304] T 00000000000000000000000000000000 P cb22b4d53b0c43f3a351d59e80cffcdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb22b4d53b0c43f3a351d59e80cffcdb; no voters:
I20260131 03:14:19.554698 9350 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cb22b4d53b0c43f3a351d59e80cffcdb [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:14:19.554667 9345 leader_election.cc:290] T 00000000000000000000000000000000 P cb22b4d53b0c43f3a351d59e80cffcdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:14:19.554836 9350 raft_consensus.cc:697] T 00000000000000000000000000000000 P cb22b4d53b0c43f3a351d59e80cffcdb [term 1 LEADER]: Becoming Leader. State: Replica: cb22b4d53b0c43f3a351d59e80cffcdb, State: Running, Role: LEADER
I20260131 03:14:19.554958 9350 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb22b4d53b0c43f3a351d59e80cffcdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb22b4d53b0c43f3a351d59e80cffcdb" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 45577 } }
I20260131 03:14:19.554970 9345 sys_catalog.cc:565] T 00000000000000000000000000000000 P cb22b4d53b0c43f3a351d59e80cffcdb [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:14:19.554832 9348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:19.550787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34809 (local address 127.19.84.62:45577)
0131 03:14:19.550977 (+ 190us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:19.550991 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:19.551019 (+ 28us) server_negotiation.cc:408] Connection header received
0131 03:14:19.551481 (+ 462us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:19.551488 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:19.551535 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:19.551678 (+ 143us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:19.552056 (+ 378us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:19.552838 (+ 782us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:19.553427 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:19.553736 (+ 309us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:19.554114 (+ 378us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:19.554123 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:19.554124 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:14:19.554317 (+ 193us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:19.554341 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:14:19.554348 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:19.554449 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:19.554542 (+ 93us) server_negotiation.cc:300] Negotiation successful
0131 03:14:19.554573 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
I20260131 03:14:19.555536 9352 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb22b4d53b0c43f3a351d59e80cffcdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb22b4d53b0c43f3a351d59e80cffcdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb22b4d53b0c43f3a351d59e80cffcdb" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 45577 } } }
I20260131 03:14:19.555630 9352 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb22b4d53b0c43f3a351d59e80cffcdb [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:19.555452 9351 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb22b4d53b0c43f3a351d59e80cffcdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb22b4d53b0c43f3a351d59e80cffcdb. Latest consensus state: current_term: 1 leader_uuid: "cb22b4d53b0c43f3a351d59e80cffcdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb22b4d53b0c43f3a351d59e80cffcdb" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 45577 } } }
I20260131 03:14:19.556133 9359 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:14:19.556198 9351 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb22b4d53b0c43f3a351d59e80cffcdb [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:19.556528 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:14:19.556717 9359 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:14:19.558259 9359 catalog_manager.cc:1357] Generated new cluster ID: ee4028aad8f640028af2dccaddd795dd
I20260131 03:14:19.558313 9359 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:14:19.561923 9348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:19.557950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37639 (local address 127.19.84.62:45577)
0131 03:14:19.557983 (+ 33us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:19.557985 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:19.558070 (+ 85us) server_negotiation.cc:408] Connection header received
0131 03:14:19.558138 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:19.558139 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:19.558150 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:19.558205 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:19.559748 (+ 1543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:19.560439 (+ 691us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:19.561115 (+ 676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:19.561282 (+ 167us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:19.561656 (+ 374us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:19.561659 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:19.561659 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:14:19.561663 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:19.561677 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:14:19.561680 (+ 3us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:19.561755 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:19.561831 (+ 76us) server_negotiation.cc:300] Negotiation successful
0131 03:14:19.561841 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260131 03:14:19.569890 9359 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:14:19.570400 9359 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:14:19.579756 9359 catalog_manager.cc:6027] T 00000000000000000000000000000000 P cb22b4d53b0c43f3a351d59e80cffcdb: Generated new TSK 0
I20260131 03:14:19.579974 9359 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:14:19.587754 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 9277
W20260131 03:14:19.592113 9259 connection.cc:537] client connection to 127.19.84.62:45577 recv error: Network error: failed to read from TLS socket (remote: 127.19.84.62:45577): Connection reset by peer (error 104)
2026-01-31T03:14:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:14:19 dist-test-slave-5nnp krb5kdc[9376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:14:19 dist-test-slave-5nnp krb5kdc[9376](info): set up 2 sockets
Jan 31 03:14:19 dist-test-slave-5nnp krb5kdc[9376](info): commencing operation
krb5kdc: starting...
W20260131 03:14:21.591876 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.986s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:14:21 dist-test-slave-5nnp krb5kdc[9376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:14:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:14:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:14:21.722035 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:46109
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:45539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:46109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:14:21.800096 9392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:14:21.800300 9392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:14:21.800326 9392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:14:21.801764 9392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:14:21.801811 9392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:14:21.801826 9392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:14:21.801839 9392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:14:21.801851 9392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:14:21.803442 9392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:45539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:46109
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:46109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:14:21.803618 9392 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:14:21.803762 9392 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:14:21.806272 9400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:21.806272 9398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:21.806409 9392 server_base.cc:1060] running on GCE node
W20260131 03:14:21.806320 9397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:21.806836 9392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:14:21.807075 9392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:14:21.808230 9392 hybrid_clock.cc:648] HybridClock initialized: now 1769829261808220 us; error 29 us; skew 500 ppm
Jan 31 03:14:21 dist-test-slave-5nnp krb5kdc[9376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829261, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:14:21.810426 9392 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:14:21.810992 9392 webserver.cc:492] Webserver started at http://127.19.84.62:45521/ using document root <none> and password file <none>
I20260131 03:14:21.811183 9392 fs_manager.cc:362] Metadata directory not provided
I20260131 03:14:21.811233 9392 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:14:21.811359 9392 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:14:21.812350 9392 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "90155238faef4f3c993bfe9ceaeb6cea"
format_stamp: "Formatted at 2026-01-31 03:14:21 on dist-test-slave-5nnp"
server_key: "00010b70c5e0635d7fb5592fb9d64d9a"
server_key_iv: "ba1237654444af4bd8a9d8d168726141"
server_key_version: "encryptionkey@0"
I20260131 03:14:21.812670 9392 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "90155238faef4f3c993bfe9ceaeb6cea"
format_stamp: "Formatted at 2026-01-31 03:14:21 on dist-test-slave-5nnp"
server_key: "00010b70c5e0635d7fb5592fb9d64d9a"
server_key_iv: "ba1237654444af4bd8a9d8d168726141"
server_key_version: "encryptionkey@0"
I20260131 03:14:21.813901 9392 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:14:21.814587 9407 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:14:21.814776 9392 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260131 03:14:21.814864 9392 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "90155238faef4f3c993bfe9ceaeb6cea"
format_stamp: "Formatted at 2026-01-31 03:14:21 on dist-test-slave-5nnp"
server_key: "00010b70c5e0635d7fb5592fb9d64d9a"
server_key_iv: "ba1237654444af4bd8a9d8d168726141"
server_key_version: "encryptionkey@0"
I20260131 03:14:21.814932 9392 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:14:21.818224 9392 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:14:21.820101 9392 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:14:21.820333 9392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:14:21.824474 9392 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:46109
I20260131 03:14:21.824518 9459 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:46109 every 8 connection(s)
I20260131 03:14:21.824834 9392 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:14:21.825404 9460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:14:21.826048 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 9392
I20260131 03:14:21.826175 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:14:21.826349 19792 external_mini_cluster.cc:1468] Setting key 2a2b215aefca4977559f730593fc67b0
I20260131 03:14:21.828804 9463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:21.826881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53925 (local address 127.19.84.62:46109)
0131 03:14:21.827056 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:21.827069 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:21.827096 (+ 27us) server_negotiation.cc:408] Connection header received
0131 03:14:21.827602 (+ 506us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:21.827611 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:21.827647 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:21.827838 (+ 191us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:21.828409 (+ 571us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:21.828417 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:21.828417 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:14:21.828422 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:21.828443 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:14:21.828452 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:21.828534 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:21.828614 (+ 80us) server_negotiation.cc:300] Negotiation successful
0131 03:14:21.828648 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260131 03:14:21.829200 9460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90155238faef4f3c993bfe9ceaeb6cea: Bootstrap starting.
I20260131 03:14:21.829759 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:14:21.829905 9460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90155238faef4f3c993bfe9ceaeb6cea: Neither blocks nor log segments found. Creating new log.
I20260131 03:14:21.830163 9460 log.cc:826] T 00000000000000000000000000000000 P 90155238faef4f3c993bfe9ceaeb6cea: Log is configured to *not* fsync() on all Append() calls
I20260131 03:14:21.830928 9460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90155238faef4f3c993bfe9ceaeb6cea: No bootstrap required, opened a new log
I20260131 03:14:21.832427 9463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:21.831362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35749 (local address 127.19.84.62:46109)
0131 03:14:21.831441 (+ 79us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:21.831443 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:21.831531 (+ 88us) server_negotiation.cc:408] Connection header received
0131 03:14:21.831558 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:21.831559 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:21.831563 (+ 4us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:21.831633 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:21.832158 (+ 525us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:21.832159 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:21.832159 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:14:21.832163 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:21.832177 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:14:21.832181 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:21.832255 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:21.832313 (+ 58us) server_negotiation.cc:300] Negotiation successful
0131 03:14:21.832327 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260131 03:14:21.832357 9460 raft_consensus.cc:359] T 00000000000000000000000000000000 P 90155238faef4f3c993bfe9ceaeb6cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90155238faef4f3c993bfe9ceaeb6cea" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 46109 } }
I20260131 03:14:21.832509 9460 raft_consensus.cc:385] T 00000000000000000000000000000000 P 90155238faef4f3c993bfe9ceaeb6cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:14:21.832540 9460 raft_consensus.cc:740] T 00000000000000000000000000000000 P 90155238faef4f3c993bfe9ceaeb6cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90155238faef4f3c993bfe9ceaeb6cea, State: Initialized, Role: FOLLOWER
I20260131 03:14:21.832638 9460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90155238faef4f3c993bfe9ceaeb6cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90155238faef4f3c993bfe9ceaeb6cea" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 46109 } }
I20260131 03:14:21.832713 9460 raft_consensus.cc:399] T 00000000000000000000000000000000 P 90155238faef4f3c993bfe9ceaeb6cea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:14:21.832746 9460 raft_consensus.cc:493] T 00000000000000000000000000000000 P 90155238faef4f3c993bfe9ceaeb6cea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:14:21.832801 9460 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 90155238faef4f3c993bfe9ceaeb6cea [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:14:21.833359 9460 raft_consensus.cc:515] T 00000000000000000000000000000000 P 90155238faef4f3c993bfe9ceaeb6cea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90155238faef4f3c993bfe9ceaeb6cea" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 46109 } }
I20260131 03:14:21.833485 9460 leader_election.cc:304] T 00000000000000000000000000000000 P 90155238faef4f3c993bfe9ceaeb6cea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90155238faef4f3c993bfe9ceaeb6cea; no voters:
I20260131 03:14:21.833639 9460 leader_election.cc:290] T 00000000000000000000000000000000 P 90155238faef4f3c993bfe9ceaeb6cea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:14:21.833689 9470 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 90155238faef4f3c993bfe9ceaeb6cea [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:14:21.833871 9460 sys_catalog.cc:565] T 00000000000000000000000000000000 P 90155238faef4f3c993bfe9ceaeb6cea [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:14:21.833858 9470 raft_consensus.cc:697] T 00000000000000000000000000000000 P 90155238faef4f3c993bfe9ceaeb6cea [term 1 LEADER]: Becoming Leader. State: Replica: 90155238faef4f3c993bfe9ceaeb6cea, State: Running, Role: LEADER
I20260131 03:14:21.835253 9470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90155238faef4f3c993bfe9ceaeb6cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90155238faef4f3c993bfe9ceaeb6cea" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 46109 } }
I20260131 03:14:21.835978 9471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90155238faef4f3c993bfe9ceaeb6cea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90155238faef4f3c993bfe9ceaeb6cea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90155238faef4f3c993bfe9ceaeb6cea" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 46109 } } }
W20260131 03:14:21.836246 9484 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 90155238faef4f3c993bfe9ceaeb6cea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260131 03:14:21.836284 9471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90155238faef4f3c993bfe9ceaeb6cea [sys.catalog]: This master's current role is: LEADER
W20260131 03:14:21.836292 9484 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:14:21.836197 9472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90155238faef4f3c993bfe9ceaeb6cea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90155238faef4f3c993bfe9ceaeb6cea. Latest consensus state: current_term: 1 leader_uuid: "90155238faef4f3c993bfe9ceaeb6cea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90155238faef4f3c993bfe9ceaeb6cea" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 46109 } } }
I20260131 03:14:21.836467 9472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90155238faef4f3c993bfe9ceaeb6cea [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:21.836472 9487 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:14:21.836776 9487 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:14:21.838147 9487 catalog_manager.cc:1357] Generated new cluster ID: 85ff79444b3e4dd093187a92b512770c
I20260131 03:14:21.838201 9487 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:14:21.860113 9487 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:14:21.860287 9487 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:14:21.871151 9487 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 90155238faef4f3c993bfe9ceaeb6cea: Generated new TSK 0
I20260131 03:14:21.871366 9487 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:14:21.876217 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 9392
2026-01-31T03:14:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2288 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:14:21 dist-test-slave-5nnp krb5kdc[9491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:14:21 dist-test-slave-5nnp krb5kdc[9491](info): set up 2 sockets
Jan 31 03:14:21 dist-test-slave-5nnp krb5kdc[9491](info): commencing operation
krb5kdc: starting...
W20260131 03:14:23.892028 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.997s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:14:23 dist-test-slave-5nnp krb5kdc[9491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:14:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:14:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:14:24.036399 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:36937
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:46531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:36937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:14:24.117148 9507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:14:24.117343 9507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:14:24.117369 9507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:14:24.118810 9507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:14:24.118853 9507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:14:24.118867 9507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:14:24.118880 9507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:14:24.118892 9507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:14:24.120596 9507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:46531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:36937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:36937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:14:24.120785 9507 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:14:24.120924 9507 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:14:24.123554 9512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:24.123677 9507 server_base.cc:1060] running on GCE node
W20260131 03:14:24.123642 9513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:24.123554 9515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:24.124043 9507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:14:24.124333 9507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:14:24.125509 9507 hybrid_clock.cc:648] HybridClock initialized: now 1769829264125490 us; error 39 us; skew 500 ppm
Jan 31 03:14:24 dist-test-slave-5nnp krb5kdc[9491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829264, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:14:24.128094 9507 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:14:24.128716 9507 webserver.cc:492] Webserver started at http://127.19.84.62:36801/ using document root <none> and password file <none>
I20260131 03:14:24.128927 9507 fs_manager.cc:362] Metadata directory not provided
I20260131 03:14:24.128984 9507 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:14:24.129114 9507 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:14:24.130120 9507 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "bbd57098a7654f8b80cfe873fb43efb7"
format_stamp: "Formatted at 2026-01-31 03:14:24 on dist-test-slave-5nnp"
server_key: "2cde76a778353ceae714ecc815207805"
server_key_iv: "cec82fa252b1e94573440d94b9b7ce79"
server_key_version: "encryptionkey@0"
I20260131 03:14:24.130477 9507 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "bbd57098a7654f8b80cfe873fb43efb7"
format_stamp: "Formatted at 2026-01-31 03:14:24 on dist-test-slave-5nnp"
server_key: "2cde76a778353ceae714ecc815207805"
server_key_iv: "cec82fa252b1e94573440d94b9b7ce79"
server_key_version: "encryptionkey@0"
I20260131 03:14:24.131800 9507 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260131 03:14:24.132576 9522 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:14:24.132782 9507 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:14:24.132865 9507 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "bbd57098a7654f8b80cfe873fb43efb7"
format_stamp: "Formatted at 2026-01-31 03:14:24 on dist-test-slave-5nnp"
server_key: "2cde76a778353ceae714ecc815207805"
server_key_iv: "cec82fa252b1e94573440d94b9b7ce79"
server_key_version: "encryptionkey@0"
I20260131 03:14:24.132920 9507 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:14:24.158773 9507 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:14:24.161069 9507 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:14:24.161182 9507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:14:24.165371 9507 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:36937
I20260131 03:14:24.165405 9574 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:36937 every 8 connection(s)
I20260131 03:14:24.165738 9507 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:14:24.166378 9575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:14:24.169083 9575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbd57098a7654f8b80cfe873fb43efb7: Bootstrap starting.
I20260131 03:14:24.169693 9575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbd57098a7654f8b80cfe873fb43efb7: Neither blocks nor log segments found. Creating new log.
I20260131 03:14:24.169965 9575 log.cc:826] T 00000000000000000000000000000000 P bbd57098a7654f8b80cfe873fb43efb7: Log is configured to *not* fsync() on all Append() calls
I20260131 03:14:24.170751 9575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbd57098a7654f8b80cfe873fb43efb7: No bootstrap required, opened a new log
I20260131 03:14:24.171176 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 9507
I20260131 03:14:24.171310 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:14:24.171546 19792 external_mini_cluster.cc:1468] Setting key 06f45c8d521f16c0cd3ec6e23f0a522f
I20260131 03:14:24.172567 9575 raft_consensus.cc:359] T 00000000000000000000000000000000 P bbd57098a7654f8b80cfe873fb43efb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd57098a7654f8b80cfe873fb43efb7" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36937 } }
I20260131 03:14:24.172717 9575 raft_consensus.cc:385] T 00000000000000000000000000000000 P bbd57098a7654f8b80cfe873fb43efb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:14:24.172748 9575 raft_consensus.cc:740] T 00000000000000000000000000000000 P bbd57098a7654f8b80cfe873fb43efb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbd57098a7654f8b80cfe873fb43efb7, State: Initialized, Role: FOLLOWER
I20260131 03:14:24.172891 9575 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbd57098a7654f8b80cfe873fb43efb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd57098a7654f8b80cfe873fb43efb7" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36937 } }
I20260131 03:14:24.172961 9575 raft_consensus.cc:399] T 00000000000000000000000000000000 P bbd57098a7654f8b80cfe873fb43efb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:14:24.172983 9575 raft_consensus.cc:493] T 00000000000000000000000000000000 P bbd57098a7654f8b80cfe873fb43efb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:14:24.173048 9575 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bbd57098a7654f8b80cfe873fb43efb7 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:14:24.173614 9575 raft_consensus.cc:515] T 00000000000000000000000000000000 P bbd57098a7654f8b80cfe873fb43efb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd57098a7654f8b80cfe873fb43efb7" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36937 } }
I20260131 03:14:24.173725 9575 leader_election.cc:304] T 00000000000000000000000000000000 P bbd57098a7654f8b80cfe873fb43efb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbd57098a7654f8b80cfe873fb43efb7; no voters:
I20260131 03:14:24.173894 9575 leader_election.cc:290] T 00000000000000000000000000000000 P bbd57098a7654f8b80cfe873fb43efb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:14:24.173974 9580 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bbd57098a7654f8b80cfe873fb43efb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:14:24.174116 9575 sys_catalog.cc:565] T 00000000000000000000000000000000 P bbd57098a7654f8b80cfe873fb43efb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:14:24.174125 9580 raft_consensus.cc:697] T 00000000000000000000000000000000 P bbd57098a7654f8b80cfe873fb43efb7 [term 1 LEADER]: Becoming Leader. State: Replica: bbd57098a7654f8b80cfe873fb43efb7, State: Running, Role: LEADER
I20260131 03:14:24.174352 9580 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbd57098a7654f8b80cfe873fb43efb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd57098a7654f8b80cfe873fb43efb7" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36937 } }
I20260131 03:14:24.174677 9580 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbd57098a7654f8b80cfe873fb43efb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbd57098a7654f8b80cfe873fb43efb7. Latest consensus state: current_term: 1 leader_uuid: "bbd57098a7654f8b80cfe873fb43efb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd57098a7654f8b80cfe873fb43efb7" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36937 } } }
I20260131 03:14:24.174685 9582 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbd57098a7654f8b80cfe873fb43efb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbd57098a7654f8b80cfe873fb43efb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd57098a7654f8b80cfe873fb43efb7" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36937 } } }
I20260131 03:14:24.174791 9580 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbd57098a7654f8b80cfe873fb43efb7 [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:24.174818 9582 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbd57098a7654f8b80cfe873fb43efb7 [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:24.175671 9588 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:14:24.176235 9588 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:14:24.178072 9588 catalog_manager.cc:1357] Generated new cluster ID: 65a6712f74b249a2a927fff42cd554d7
I20260131 03:14:24.178126 9588 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Jan 31 03:14:24 dist-test-slave-5nnp krb5kdc[9491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:14:24.182188 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:24.172115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59365 (local address 127.19.84.62:36937)
0131 03:14:24.172327 (+ 212us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:24.172338 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:24.172365 (+ 27us) server_negotiation.cc:408] Connection header received
0131 03:14:24.172518 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:24.172524 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:24.172591 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:24.172728 (+ 137us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:24.173637 (+ 909us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:24.174881 (+ 1244us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:24.176002 (+ 1121us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:24.176255 (+ 253us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:24.178658 (+ 2403us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:24.178687 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:24.178688 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:24.178717 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:24.180426 (+ 1709us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:24.180796 (+ 370us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:24.180800 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:24.180800 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:24.180861 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:24.181030 (+ 169us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:24.181031 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:24.181031 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:24.181361 (+ 330us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:24.181545 (+ 184us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:24.181816 (+ 271us) server_negotiation.cc:300] Negotiation successful
0131 03:14:24.181960 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":113,"threads_started":1}
I20260131 03:14:24.183224 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:14:24.190953 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:24.184822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55099 (local address 127.19.84.62:36937)
0131 03:14:24.184877 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:24.184879 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:24.185013 (+ 134us) server_negotiation.cc:408] Connection header received
0131 03:14:24.185094 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:24.185095 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:24.185114 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:24.185181 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:24.185862 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:24.186325 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:24.186851 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:24.187082 (+ 231us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:24.188039 (+ 957us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:24.188043 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:24.188043 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:24.188051 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:24.189872 (+ 1821us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:24.190194 (+ 322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:24.190195 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:24.190195 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:24.190226 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:24.190420 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:24.190421 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:24.190421 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:24.190516 (+ 95us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:24.190620 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:24.190730 (+ 110us) server_negotiation.cc:300] Negotiation successful
0131 03:14:24.190824 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260131 03:14:24.206997 9588 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:14:24.207702 9588 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:14:24.216301 9588 catalog_manager.cc:6027] T 00000000000000000000000000000000 P bbd57098a7654f8b80cfe873fb43efb7: Generated new TSK 0
I20260131 03:14:24.216542 9588 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:14:24.220899 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 9507
2026-01-31T03:14:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:14:24 dist-test-slave-5nnp krb5kdc[9606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:14:24 dist-test-slave-5nnp krb5kdc[9606](info): set up 2 sockets
Jan 31 03:14:24 dist-test-slave-5nnp krb5kdc[9606](info): commencing operation
krb5kdc: starting...
W20260131 03:14:26.235972 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.996s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:14:26 dist-test-slave-5nnp krb5kdc[9606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:14:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:14:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:14:26.372026 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:42791
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:44501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:42791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:14:26.450639 9622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:14:26.450830 9622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:14:26.450855 9622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:14:26.452379 9622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:14:26.452428 9622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:14:26.452443 9622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:14:26.452455 9622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:14:26.452467 9622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:14:26.454088 9622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:44501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:42791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:42791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:14:26.454309 9622 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:14:26.454488 9622 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:14:26.457160 9627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:26.457257 9630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:26.457322 9622 server_base.cc:1060] running on GCE node
W20260131 03:14:26.457165 9628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:26.457655 9622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:14:26.457911 9622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:14:26.459084 9622 hybrid_clock.cc:648] HybridClock initialized: now 1769829266459058 us; error 42 us; skew 500 ppm
Jan 31 03:14:26 dist-test-slave-5nnp krb5kdc[9606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829266, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:14:26.461684 9622 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:14:26.462316 9622 webserver.cc:492] Webserver started at http://127.19.84.62:41761/ using document root <none> and password file <none>
I20260131 03:14:26.462517 9622 fs_manager.cc:362] Metadata directory not provided
I20260131 03:14:26.462572 9622 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:14:26.462709 9622 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:14:26.463757 9622 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "72a72ce3bfc04d4d865775763da37a62"
format_stamp: "Formatted at 2026-01-31 03:14:26 on dist-test-slave-5nnp"
server_key: "8b759b4f5b3094eb15713503cb2f3f07"
server_key_iv: "68734beb6d6e7500f4d5a735289b65c0"
server_key_version: "encryptionkey@0"
I20260131 03:14:26.464207 9622 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "72a72ce3bfc04d4d865775763da37a62"
format_stamp: "Formatted at 2026-01-31 03:14:26 on dist-test-slave-5nnp"
server_key: "8b759b4f5b3094eb15713503cb2f3f07"
server_key_iv: "68734beb6d6e7500f4d5a735289b65c0"
server_key_version: "encryptionkey@0"
I20260131 03:14:26.465665 9622 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260131 03:14:26.466478 9637 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:14:26.466655 9622 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:14:26.466743 9622 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "72a72ce3bfc04d4d865775763da37a62"
format_stamp: "Formatted at 2026-01-31 03:14:26 on dist-test-slave-5nnp"
server_key: "8b759b4f5b3094eb15713503cb2f3f07"
server_key_iv: "68734beb6d6e7500f4d5a735289b65c0"
server_key_version: "encryptionkey@0"
I20260131 03:14:26.466827 9622 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:14:26.478914 9622 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:14:26.484623 9622 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:14:26.484794 9622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:14:26.489208 9622 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:42791
I20260131 03:14:26.489241 9689 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:42791 every 8 connection(s)
I20260131 03:14:26.489573 9622 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:14:26.490218 9690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:14:26.492781 9690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72a72ce3bfc04d4d865775763da37a62: Bootstrap starting.
I20260131 03:14:26.493361 9690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72a72ce3bfc04d4d865775763da37a62: Neither blocks nor log segments found. Creating new log.
I20260131 03:14:26.493613 9690 log.cc:826] T 00000000000000000000000000000000 P 72a72ce3bfc04d4d865775763da37a62: Log is configured to *not* fsync() on all Append() calls
I20260131 03:14:26.494331 9690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72a72ce3bfc04d4d865775763da37a62: No bootstrap required, opened a new log
I20260131 03:14:26.495954 9690 raft_consensus.cc:359] T 00000000000000000000000000000000 P 72a72ce3bfc04d4d865775763da37a62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a72ce3bfc04d4d865775763da37a62" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42791 } }
I20260131 03:14:26.496078 9690 raft_consensus.cc:385] T 00000000000000000000000000000000 P 72a72ce3bfc04d4d865775763da37a62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:14:26.496098 9690 raft_consensus.cc:740] T 00000000000000000000000000000000 P 72a72ce3bfc04d4d865775763da37a62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72a72ce3bfc04d4d865775763da37a62, State: Initialized, Role: FOLLOWER
I20260131 03:14:26.496250 9690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72a72ce3bfc04d4d865775763da37a62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a72ce3bfc04d4d865775763da37a62" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42791 } }
I20260131 03:14:26.496308 9690 raft_consensus.cc:399] T 00000000000000000000000000000000 P 72a72ce3bfc04d4d865775763da37a62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:14:26.496330 9690 raft_consensus.cc:493] T 00000000000000000000000000000000 P 72a72ce3bfc04d4d865775763da37a62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:14:26.496390 9690 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 72a72ce3bfc04d4d865775763da37a62 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:14:26.496869 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 9622
I20260131 03:14:26.496992 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:14:26.497123 9690 raft_consensus.cc:515] T 00000000000000000000000000000000 P 72a72ce3bfc04d4d865775763da37a62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a72ce3bfc04d4d865775763da37a62" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42791 } }
I20260131 03:14:26.497193 19792 external_mini_cluster.cc:1468] Setting key a15fb165711abec13f5b1f29e105152d
I20260131 03:14:26.497251 9690 leader_election.cc:304] T 00000000000000000000000000000000 P 72a72ce3bfc04d4d865775763da37a62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72a72ce3bfc04d4d865775763da37a62; no voters:
I20260131 03:14:26.497397 9690 leader_election.cc:290] T 00000000000000000000000000000000 P 72a72ce3bfc04d4d865775763da37a62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:14:26.497430 9693 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 72a72ce3bfc04d4d865775763da37a62 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:14:26.497527 9693 raft_consensus.cc:697] T 00000000000000000000000000000000 P 72a72ce3bfc04d4d865775763da37a62 [term 1 LEADER]: Becoming Leader. State: Replica: 72a72ce3bfc04d4d865775763da37a62, State: Running, Role: LEADER
I20260131 03:14:26.497628 9693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72a72ce3bfc04d4d865775763da37a62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a72ce3bfc04d4d865775763da37a62" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42791 } }
I20260131 03:14:26.497637 9690 sys_catalog.cc:565] T 00000000000000000000000000000000 P 72a72ce3bfc04d4d865775763da37a62 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:14:26.498075 9695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72a72ce3bfc04d4d865775763da37a62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72a72ce3bfc04d4d865775763da37a62. Latest consensus state: current_term: 1 leader_uuid: "72a72ce3bfc04d4d865775763da37a62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a72ce3bfc04d4d865775763da37a62" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42791 } } }
I20260131 03:14:26.498099 9694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72a72ce3bfc04d4d865775763da37a62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72a72ce3bfc04d4d865775763da37a62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a72ce3bfc04d4d865775763da37a62" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 42791 } } }
I20260131 03:14:26.498229 9694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72a72ce3bfc04d4d865775763da37a62 [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:26.498617 9699 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:14:26.499296 9699 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:14:26.498216 9695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72a72ce3bfc04d4d865775763da37a62 [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:26.500962 9699 catalog_manager.cc:1357] Generated new cluster ID: 5174263e6a0044649c56001234765541
I20260131 03:14:26.501016 9699 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:14:26.503216 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:26.497789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40155 (local address 127.19.84.62:42791)
0131 03:14:26.499435 (+ 1646us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:26.499448 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:26.499480 (+ 32us) server_negotiation.cc:408] Connection header received
0131 03:14:26.499596 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:26.499602 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:26.499658 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:26.499817 (+ 159us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:26.500287 (+ 470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:26.501330 (+ 1043us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:26.501914 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:26.502182 (+ 268us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:26.502611 (+ 429us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:26.502632 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:26.502633 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:14:26.502818 (+ 185us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:26.502835 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:14:26.502843 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:26.502927 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:26.502998 (+ 71us) server_negotiation.cc:300] Negotiation successful
0131 03:14:26.503045 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1599,"thread_start_us":65,"threads_started":1}
I20260131 03:14:26.504292 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:14:26.509848 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:26.505642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42479 (local address 127.19.84.62:42791)
0131 03:14:26.505680 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:26.505682 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:26.507573 (+ 1891us) server_negotiation.cc:408] Connection header received
0131 03:14:26.507668 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:26.507669 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:26.507689 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:26.507752 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:26.508006 (+ 254us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:26.508439 (+ 433us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:26.509004 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:26.509527 (+ 523us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:26.509569 (+ 42us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:26.509571 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:26.509572 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:14:26.509576 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:26.509585 (+ 9us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:14:26.509589 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:26.509660 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:26.509738 (+ 78us) server_negotiation.cc:300] Negotiation successful
0131 03:14:26.509762 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260131 03:14:26.538342 9699 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:14:26.539141 9699 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:14:26.546947 9699 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 72a72ce3bfc04d4d865775763da37a62: Generated new TSK 0
I20260131 03:14:26.547163 9699 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:14:26.557807 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 9622
2026-01-31T03:14:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:14:26 dist-test-slave-5nnp krb5kdc[9721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:14:26 dist-test-slave-5nnp krb5kdc[9721](info): set up 2 sockets
Jan 31 03:14:26 dist-test-slave-5nnp krb5kdc[9721](info): commencing operation
krb5kdc: starting...
W20260131 03:14:28.556002 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.979s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:14:28 dist-test-slave-5nnp krb5kdc[9721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:14:28Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:14:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:14:28.688442 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:38679
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:36897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:38679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:14:28.766269 9737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:14:28.766489 9737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:14:28.766528 9737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:14:28.768030 9737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:14:28.768108 9737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:14:28.768141 9737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:14:28.768162 9737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:14:28.768193 9737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:14:28.769778 9737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:36897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:38679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:38679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:14:28.770040 9737 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:14:28.770237 9737 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:14:28.772783 9742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:28.772746 9745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:28.772746 9743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:28.773015 9737 server_base.cc:1060] running on GCE node
I20260131 03:14:28.773219 9737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:14:28.773461 9737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:14:28.774610 9737 hybrid_clock.cc:648] HybridClock initialized: now 1769829268774592 us; error 30 us; skew 500 ppm
Jan 31 03:14:28 dist-test-slave-5nnp krb5kdc[9721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829268, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:14:28.776823 9737 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:14:28.777393 9737 webserver.cc:492] Webserver started at http://127.19.84.62:41953/ using document root <none> and password file <none>
I20260131 03:14:28.777547 9737 fs_manager.cc:362] Metadata directory not provided
I20260131 03:14:28.777578 9737 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:14:28.777657 9737 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:14:28.778555 9737 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "b75f3a67d7f64f48bb8b28b361194159"
format_stamp: "Formatted at 2026-01-31 03:14:28 on dist-test-slave-5nnp"
server_key: "aa7a2add99092be4ce8ffc2b36e2e88c"
server_key_iv: "0231f59859475669775068be83e92c35"
server_key_version: "encryptionkey@0"
I20260131 03:14:28.778810 9737 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "b75f3a67d7f64f48bb8b28b361194159"
format_stamp: "Formatted at 2026-01-31 03:14:28 on dist-test-slave-5nnp"
server_key: "aa7a2add99092be4ce8ffc2b36e2e88c"
server_key_iv: "0231f59859475669775068be83e92c35"
server_key_version: "encryptionkey@0"
I20260131 03:14:28.780054 9737 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260131 03:14:28.780722 9752 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:14:28.780957 9737 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:14:28.781054 9737 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "b75f3a67d7f64f48bb8b28b361194159"
format_stamp: "Formatted at 2026-01-31 03:14:28 on dist-test-slave-5nnp"
server_key: "aa7a2add99092be4ce8ffc2b36e2e88c"
server_key_iv: "0231f59859475669775068be83e92c35"
server_key_version: "encryptionkey@0"
I20260131 03:14:28.781131 9737 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:14:28.784416 9737 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:14:28.786428 9737 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:14:28.786561 9737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:14:28.790656 9737 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:38679
I20260131 03:14:28.790689 9804 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:38679 every 8 connection(s)
I20260131 03:14:28.791007 9737 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:14:28.791747 9805 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:14:28.792498 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 9737
I20260131 03:14:28.792599 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:14:28.792800 19792 external_mini_cluster.cc:1468] Setting key 805000f7b32301cee4a5d6011cc8c2a6
I20260131 03:14:28.795130 9808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:28.793367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40361 (local address 127.19.84.62:38679)
0131 03:14:28.793517 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:28.793529 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:28.793553 (+ 24us) server_negotiation.cc:408] Connection header received
0131 03:14:28.793935 (+ 382us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:28.793944 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:28.793979 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:28.794137 (+ 158us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:28.794748 (+ 611us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:28.794751 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:28.794751 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:14:28.794755 (+ 4us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:28.794780 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:14:28.794788 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:28.794888 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:28.794949 (+ 61us) server_negotiation.cc:300] Negotiation successful
0131 03:14:28.794983 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260131 03:14:28.795778 9805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b75f3a67d7f64f48bb8b28b361194159: Bootstrap starting.
I20260131 03:14:28.796110 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:14:28.796499 9805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b75f3a67d7f64f48bb8b28b361194159: Neither blocks nor log segments found. Creating new log.
I20260131 03:14:28.796783 9805 log.cc:826] T 00000000000000000000000000000000 P b75f3a67d7f64f48bb8b28b361194159: Log is configured to *not* fsync() on all Append() calls
I20260131 03:14:28.797577 9805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b75f3a67d7f64f48bb8b28b361194159: No bootstrap required, opened a new log
I20260131 03:14:28.798580 9805 raft_consensus.cc:359] T 00000000000000000000000000000000 P b75f3a67d7f64f48bb8b28b361194159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75f3a67d7f64f48bb8b28b361194159" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 38679 } }
I20260131 03:14:28.798696 9805 raft_consensus.cc:385] T 00000000000000000000000000000000 P b75f3a67d7f64f48bb8b28b361194159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260131 03:14:28.798717 9805 raft_consensus.cc:740] T 00000000000000000000000000000000 P b75f3a67d7f64f48bb8b28b361194159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75f3a67d7f64f48bb8b28b361194159, State: Initialized, Role: FOLLOWER
I20260131 03:14:28.798815 9805 consensus_queue.cc:260] T 00000000000000000000000000000000 P b75f3a67d7f64f48bb8b28b361194159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75f3a67d7f64f48bb8b28b361194159" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 38679 } }
I20260131 03:14:28.798882 9805 raft_consensus.cc:399] T 00000000000000000000000000000000 P b75f3a67d7f64f48bb8b28b361194159 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260131 03:14:28.798902 9805 raft_consensus.cc:493] T 00000000000000000000000000000000 P b75f3a67d7f64f48bb8b28b361194159 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260131 03:14:28.798946 9805 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b75f3a67d7f64f48bb8b28b361194159 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:14:28.799517 9805 raft_consensus.cc:515] T 00000000000000000000000000000000 P b75f3a67d7f64f48bb8b28b361194159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75f3a67d7f64f48bb8b28b361194159" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 38679 } }
I20260131 03:14:28.799639 9805 leader_election.cc:304] T 00000000000000000000000000000000 P b75f3a67d7f64f48bb8b28b361194159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b75f3a67d7f64f48bb8b28b361194159; no voters:
I20260131 03:14:28.799789 9805 leader_election.cc:290] T 00000000000000000000000000000000 P b75f3a67d7f64f48bb8b28b361194159 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260131 03:14:28.799849 9815 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b75f3a67d7f64f48bb8b28b361194159 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:14:28.800020 9815 raft_consensus.cc:697] T 00000000000000000000000000000000 P b75f3a67d7f64f48bb8b28b361194159 [term 1 LEADER]: Becoming Leader. State: Replica: b75f3a67d7f64f48bb8b28b361194159, State: Running, Role: LEADER
I20260131 03:14:28.800078 9805 sys_catalog.cc:565] T 00000000000000000000000000000000 P b75f3a67d7f64f48bb8b28b361194159 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:14:28.800196 9815 consensus_queue.cc:237] T 00000000000000000000000000000000 P b75f3a67d7f64f48bb8b28b361194159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75f3a67d7f64f48bb8b28b361194159" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 38679 } }
I20260131 03:14:28.800595 9816 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75f3a67d7f64f48bb8b28b361194159 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b75f3a67d7f64f48bb8b28b361194159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75f3a67d7f64f48bb8b28b361194159" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 38679 } } }
I20260131 03:14:28.800673 9816 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75f3a67d7f64f48bb8b28b361194159 [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:28.800812 9816 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75f3a67d7f64f48bb8b28b361194159 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b75f3a67d7f64f48bb8b28b361194159. Latest consensus state: current_term: 1 leader_uuid: "b75f3a67d7f64f48bb8b28b361194159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75f3a67d7f64f48bb8b28b361194159" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 38679 } } }
I20260131 03:14:28.800877 9816 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75f3a67d7f64f48bb8b28b361194159 [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:28.801004 9808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:28.799706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35329 (local address 127.19.84.62:38679)
0131 03:14:28.799743 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:28.799745 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:28.799877 (+ 132us) server_negotiation.cc:408] Connection header received
0131 03:14:28.799983 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:28.799984 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:28.799990 (+ 6us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:28.800020 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:28.800755 (+ 735us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:28.800756 (+ 1us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:28.800756 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0131 03:14:28.800761 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:28.800777 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0131 03:14:28.800781 (+ 4us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:28.800813 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:28.800931 (+ 118us) server_negotiation.cc:300] Negotiation successful
0131 03:14:28.800946 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260131 03:14:28.801302 9821 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:14:28.801808 9821 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:14:28.803226 9821 catalog_manager.cc:1357] Generated new cluster ID: cf52a5cc709d436cb1a28d2ca38df8d2
I20260131 03:14:28.803297 9821 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:14:28.815516 9821 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:14:28.815706 9821 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:14:28.825145 9821 catalog_manager.cc:6027] T 00000000000000000000000000000000 P b75f3a67d7f64f48bb8b28b361194159: Generated new TSK 0
I20260131 03:14:28.825356 9821 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260131 03:14:28.829563 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 9737
2026-01-31T03:14:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2271 ms)
[----------] 18 tests from AuthTokenIssuingTest (41465 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:14:28 dist-test-slave-5nnp krb5kdc[9836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:14:28 dist-test-slave-5nnp krb5kdc[9836](info): set up 2 sockets
Jan 31 03:14:28 dist-test-slave-5nnp krb5kdc[9836](info): commencing operation
krb5kdc: starting...
W20260131 03:14:30.864075 19792 mini_kdc.cc:121] Time spent starting KDC: real 2.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:14:30 dist-test-slave-5nnp krb5kdc[9836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:14:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:14:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:14:31.002082 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:36877
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:36877,127.19.84.61:40779,127.19.84.60:35033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:14:31.084120 9852 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260131 03:14:31.084311 9852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:14:31.084327 9852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:14:31.084347 9852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:14:31.085704 9852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:14:31.085741 9852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:14:31.085757 9852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:14:31.085768 9852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:14:31.085780 9852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:14:31.087281 9852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:36877,127.19.84.61:40779,127.19.84.60:35033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:36877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:14:31.087503 9852 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:14:31.087648 9852 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:14:31.089988 9860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:31.089990 9858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:31.090175 9852 server_base.cc:1060] running on GCE node
W20260131 03:14:31.090086 9857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:31.090488 9852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:14:31.090736 9852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:14:31.092059 9852 hybrid_clock.cc:648] HybridClock initialized: now 1769829271092029 us; error 39 us; skew 500 ppm
Jan 31 03:14:31 dist-test-slave-5nnp krb5kdc[9836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829271, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:14:31.094355 9852 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:14:31.094933 9852 webserver.cc:492] Webserver started at http://127.19.84.62:41849/ using document root <none> and password file <none>
I20260131 03:14:31.095136 9852 fs_manager.cc:362] Metadata directory not provided
I20260131 03:14:31.095173 9852 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:14:31.095297 9852 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:14:31.096287 9852 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "fd7d457eb5dc43e897a21b81dfcb01f1"
format_stamp: "Formatted at 2026-01-31 03:14:31 on dist-test-slave-5nnp"
server_key: "fba2cd0637cda8acf561d0e93579a350"
server_key_iv: "1a87d76eff03a18c1c2adf6fca7ae36f"
server_key_version: "encryptionkey@0"
I20260131 03:14:31.096601 9852 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "fd7d457eb5dc43e897a21b81dfcb01f1"
format_stamp: "Formatted at 2026-01-31 03:14:31 on dist-test-slave-5nnp"
server_key: "fba2cd0637cda8acf561d0e93579a350"
server_key_iv: "1a87d76eff03a18c1c2adf6fca7ae36f"
server_key_version: "encryptionkey@0"
I20260131 03:14:31.097793 9852 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260131 03:14:31.098534 9867 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:14:31.098728 9852 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:14:31.098816 9852 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "fd7d457eb5dc43e897a21b81dfcb01f1"
format_stamp: "Formatted at 2026-01-31 03:14:31 on dist-test-slave-5nnp"
server_key: "fba2cd0637cda8acf561d0e93579a350"
server_key_iv: "1a87d76eff03a18c1c2adf6fca7ae36f"
server_key_version: "encryptionkey@0"
I20260131 03:14:31.098925 9852 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:14:31.105872 9852 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:14:31.107796 9852 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:14:31.107949 9852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:14:31.111886 9852 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:36877
I20260131 03:14:31.111922 9919 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:36877 every 8 connection(s)
I20260131 03:14:31.112366 9852 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:14:31.112970 9920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:14:31.114786 9920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36877 } has no permanent_uuid. Determining permanent_uuid...
I20260131 03:14:31.116580 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 9852
I20260131 03:14:31.116712 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:14:31.116942 19792 external_mini_cluster.cc:1468] Setting key d188e72c1de78286df4bfac31f53897a
Jan 31 03:14:31 dist-test-slave-5nnp krb5kdc[9836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829271, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.84.61@KRBTEST.COM; defaulting to no policy
I20260131 03:14:31.123956 9922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.115418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:36877 (local address 127.0.0.1:57858)
0131 03:14:31.115610 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.115641 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0131 03:14:31.116042 (+ 401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:14:31.116345 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:14:31.116348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:14:31.116767 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:14:31.117144 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:31.117148 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.118201 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:14:31.118202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:14:31.118767 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:31.118769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.118904 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:31.119505 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:14:31.119521 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:14:31.120891 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:14:31.122637 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:31.122642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:31.122645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:31.122822 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:31.123039 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:31.123040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:31.123041 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:31.123094 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:31.123417 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:14:31.123421 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:14:31.123488 (+ 67us) client_negotiation.cc:769] Sending connection context
0131 03:14:31.123581 (+ 93us) client_negotiation.cc:240] Negotiation successful
0131 03:14:31.123822 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":128,"thread_start_us":94,"threads_started":1}
I20260131 03:14:31.123955 9923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.115437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57858 (local address 127.19.84.62:36877)
0131 03:14:31.115616 (+ 179us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.115627 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.115732 (+ 105us) server_negotiation.cc:408] Connection header received
0131 03:14:31.116179 (+ 447us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.116188 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.116221 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.116302 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.117254 (+ 952us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.118081 (+ 827us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:31.118869 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.119079 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:31.120983 (+ 1904us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:31.120991 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:31.120992 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:31.120997 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:31.122538 (+ 1541us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.122926 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.122929 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.122929 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.122974 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.123153 (+ 179us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.123154 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.123155 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.123323 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:31.123439 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:31.123612 (+ 173us) server_negotiation.cc:300] Negotiation successful
0131 03:14:31.123759 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":104,"threads_started":1}
I20260131 03:14:31.124801 9920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 } has no permanent_uuid. Determining permanent_uuid...
I20260131 03:14:31.125128 9922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.125021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:40779 (local address 127.0.0.1:34574)
0131 03:14:31.125064 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.125105 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.61:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260131 03:14:31.125247 9869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.84.61:40779: connect: Connection refused (error 111)
I20260131 03:14:31.125658 9922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.125579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:40779 (local address 127.0.0.1:34576)
0131 03:14:31.125618 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.125642 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.61:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260131 03:14:31.125766 9920 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.61:40779: Network error: Client connection negotiation failed: client connection to 127.19.84.61:40779: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:14:31.158171 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.61:40779
--webserver_interface=127.19.84.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:36877,127.19.84.61:40779,127.19.84.60:35033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260131 03:14:31.174964 9920 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 } attempt: 1
I20260131 03:14:31.175475 9922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.175313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:40779 (local address 127.0.0.1:34582)
0131 03:14:31.175390 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.175436 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.61:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260131 03:14:31.175846 9922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.175743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:40779 (local address 127.0.0.1:34588)
0131 03:14:31.175788 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.175814 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.61:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260131 03:14:31.175995 9920 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.61:40779: Network error: Client connection negotiation failed: client connection to 127.19.84.61:40779: connect: Connection refused (error 111)
W20260131 03:14:31.236780 9929 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260131 03:14:31.236985 9929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:14:31.237002 9929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:14:31.237025 9929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:14:31.238552 9929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:14:31.238601 9929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:14:31.238615 9929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:14:31.238628 9929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:14:31.238641 9929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:14:31.240317 9929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:36877,127.19.84.61:40779,127.19.84.60:35033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.61:40779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.84.61
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:14:31.240574 9929 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:14:31.240775 9929 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:14:31.243403 9934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:31.243408 9937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:31.243598 9935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:31.243522 9929 server_base.cc:1060] running on GCE node
I20260131 03:14:31.243853 9929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:14:31.244120 9929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:14:31.244167 9920 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 } attempt: 2
I20260131 03:14:31.244565 9922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.244454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:40779 (local address 127.0.0.1:34590)
0131 03:14:31.244499 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.244531 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.61:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260131 03:14:31.244870 9922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.244792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:40779 (local address 127.0.0.1:34598)
0131 03:14:31.244831 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.244853 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.61:40779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260131 03:14:31.245046 9920 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.61:40779: Network error: Client connection negotiation failed: client connection to 127.19.84.61:40779: connect: Connection refused (error 111)
I20260131 03:14:31.245283 9929 hybrid_clock.cc:648] HybridClock initialized: now 1769829271245265 us; error 37 us; skew 500 ppm
Jan 31 03:14:31 dist-test-slave-5nnp krb5kdc[9836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829271, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:14:31.247546 9929 init.cc:377] Logged in from keytab as kudu/127.19.84.61@KRBTEST.COM (short username kudu)
I20260131 03:14:31.248080 9929 webserver.cc:492] Webserver started at http://127.19.84.61:40187/ using document root <none> and password file <none>
I20260131 03:14:31.248253 9929 fs_manager.cc:362] Metadata directory not provided
I20260131 03:14:31.248286 9929 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:14:31.248379 9929 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:14:31.249341 9929 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-1/data/instance:
uuid: "46770edc85f54c77a05592a6b729c63c"
format_stamp: "Formatted at 2026-01-31 03:14:31 on dist-test-slave-5nnp"
server_key: "f982eaff57230a6f1971f6b52efd288b"
server_key_iv: "073cbb2e3a07f295ee5fa1b7a5191013"
server_key_version: "encryptionkey@0"
I20260131 03:14:31.249657 9929 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-1/wal/instance:
uuid: "46770edc85f54c77a05592a6b729c63c"
format_stamp: "Formatted at 2026-01-31 03:14:31 on dist-test-slave-5nnp"
server_key: "f982eaff57230a6f1971f6b52efd288b"
server_key_iv: "073cbb2e3a07f295ee5fa1b7a5191013"
server_key_version: "encryptionkey@0"
I20260131 03:14:31.250841 9929 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260131 03:14:31.251559 9944 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:14:31.251736 9929 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260131 03:14:31.251817 9929 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-1/wal
uuid: "46770edc85f54c77a05592a6b729c63c"
format_stamp: "Formatted at 2026-01-31 03:14:31 on dist-test-slave-5nnp"
server_key: "f982eaff57230a6f1971f6b52efd288b"
server_key_iv: "073cbb2e3a07f295ee5fa1b7a5191013"
server_key_version: "encryptionkey@0"
I20260131 03:14:31.251896 9929 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:14:31.281863 9929 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:14:31.283689 9929 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:14:31.283838 9929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:14:31.287840 9929 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.61:40779
I20260131 03:14:31.287885 9996 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.61:40779 every 8 connection(s)
I20260131 03:14:31.288231 9929 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-1/data/info.pb
I20260131 03:14:31.288903 9997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:14:31.290699 9997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36877 } has no permanent_uuid. Determining permanent_uuid...
I20260131 03:14:31.292843 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 9929
I20260131 03:14:31.292961 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-1/wal/instance
I20260131 03:14:31.293119 19792 external_mini_cluster.cc:1468] Setting key d3a8c0d57d092045335bdc9f04d702a1
Jan 31 03:14:31 dist-test-slave-5nnp krb5kdc[9836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829271, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.61@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:14:31.299402 9923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.291409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53646 (local address 127.19.84.62:36877)
0131 03:14:31.291469 (+ 60us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.291471 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.291670 (+ 199us) server_negotiation.cc:408] Connection header received
0131 03:14:31.291985 (+ 315us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.291986 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.292000 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.292062 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.292783 (+ 721us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.293476 (+ 693us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:31.294043 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.294248 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:31.296387 (+ 2139us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:31.296392 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:31.296393 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:31.296401 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:31.298140 (+ 1739us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.298521 (+ 381us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.298522 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.298523 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.298552 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.298734 (+ 182us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.298734 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.298735 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.298860 (+ 125us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:31.298933 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:31.299168 (+ 235us) server_negotiation.cc:300] Negotiation successful
0131 03:14:31.299267 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260131 03:14:31.299362 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.291347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:36877 (local address 127.0.0.1:53646)
0131 03:14:31.291527 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.291567 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0131 03:14:31.291891 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:14:31.292134 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:14:31.292137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:14:31.292413 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:14:31.292685 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:31.292688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.293571 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:14:31.293572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:14:31.293965 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:31.293966 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.294058 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:31.294599 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:14:31.294618 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:14:31.296255 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:14:31.298239 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:31.298245 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:31.298247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:31.298444 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:31.298619 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:31.298619 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:31.298620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:31.298676 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:31.298926 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:14:31.298930 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:14:31.299036 (+ 106us) client_negotiation.cc:769] Sending connection context
0131 03:14:31.299119 (+ 83us) client_negotiation.cc:240] Negotiation successful
0131 03:14:31.299220 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":100,"threads_started":1}
I20260131 03:14:31.300053 9997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.19.84.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.60@KRBTEST.COM" created.
Jan 31 03:14:31 dist-test-slave-5nnp krb5kdc[9836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829271, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.61@KRBTEST.COM for kudu/127.19.84.61@KRBTEST.COM
I20260131 03:14:31.307279 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.300338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:40779 (local address 127.0.0.1:54128)
0131 03:14:31.300559 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.300572 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:14:31.300628 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:14:31.300773 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:14:31.300774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:14:31.300917 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:14:31.301098 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:31.301100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.301918 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:14:31.301919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:14:31.302390 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:31.302392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.302447 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:31.302946 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:14:31.302950 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:14:31.304514 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:14:31.306356 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:31.306357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:31.306358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:31.306497 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:31.306687 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:31.306688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:31.306688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:31.306715 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:31.306951 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:14:31.306952 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:14:31.306989 (+ 37us) client_negotiation.cc:769] Sending connection context
0131 03:14:31.307050 (+ 61us) client_negotiation.cc:240] Negotiation successful
0131 03:14:31.307148 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203}
I20260131 03:14:31.307361 10001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.300412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54128 (local address 127.19.84.61:40779)
0131 03:14:31.300556 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.300563 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.300617 (+ 54us) server_negotiation.cc:408] Connection header received
0131 03:14:31.300688 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.300690 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.300700 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.300752 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.301194 (+ 442us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.301843 (+ 649us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:31.302463 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.302781 (+ 318us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:31.304609 (+ 1828us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:31.304618 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:31.304618 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:31.304625 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:31.306256 (+ 1631us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.306583 (+ 327us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.306586 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.306587 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.306623 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.306769 (+ 146us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.306770 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.306770 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.306874 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:31.306972 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:31.307075 (+ 103us) server_negotiation.cc:300] Negotiation successful
0131 03:14:31.307217 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":121,"thread_start_us":100,"threads_started":1}
I20260131 03:14:31.307889 9997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } has no permanent_uuid. Determining permanent_uuid...
I20260131 03:14:31.308251 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.308128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:35033 (local address 127.0.0.1:57318)
0131 03:14:31.308179 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.308221 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260131 03:14:31.308390 9945 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.84.60:35033: connect: Connection refused (error 111)
I20260131 03:14:31.308794 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.308733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:35033 (local address 127.0.0.1:57326)
0131 03:14:31.308757 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.308779 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260131 03:14:31.308928 9997 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.60:35033: Network error: Client connection negotiation failed: client connection to 127.19.84.60:35033: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.60@KRBTEST.COM" created.
I20260131 03:14:31.336205 9920 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:14:31.338770 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.60:35033
--webserver_interface=127.19.84.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:36877,127.19.84.61:40779,127.19.84.60:35033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jan 31 03:14:31 dist-test-slave-5nnp krb5kdc[9836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829271, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for kudu/127.19.84.61@KRBTEST.COM
I20260131 03:14:31.343155 9922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.336581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:40779 (local address 127.0.0.1:54134)
0131 03:14:31.336667 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.336681 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0131 03:14:31.336763 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:14:31.336947 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:14:31.336948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:14:31.337149 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:14:31.337332 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:31.337334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.337959 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:14:31.337960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:14:31.338432 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:31.338435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.338516 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:31.339040 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:14:31.339047 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:14:31.340459 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:14:31.342327 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:31.342328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:31.342329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:31.342472 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:31.342643 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:31.342644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:31.342644 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:31.342668 (+ 24us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:31.342909 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:14:31.342910 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:14:31.342951 (+ 41us) client_negotiation.cc:769] Sending connection context
0131 03:14:31.343007 (+ 56us) client_negotiation.cc:240] Negotiation successful
0131 03:14:31.343084 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260131 03:14:31.343288 10001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.336671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54134 (local address 127.19.84.61:40779)
0131 03:14:31.336733 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.336736 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.336747 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:14:31.336854 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.336855 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.336870 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.336939 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.337413 (+ 474us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.337877 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:31.338520 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.338683 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:31.340550 (+ 1867us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:31.340554 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:31.340554 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:31.340561 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:31.342247 (+ 1686us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.342543 (+ 296us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.342545 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.342545 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.342575 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.342709 (+ 134us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.342710 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.342710 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.342841 (+ 131us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:31.342905 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:31.343057 (+ 152us) server_negotiation.cc:300] Negotiation successful
0131 03:14:31.343153 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260131 03:14:31.343732 9920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } has no permanent_uuid. Determining permanent_uuid...
I20260131 03:14:31.344085 9922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.343981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:35033 (local address 127.0.0.1:57340)
0131 03:14:31.344030 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.344065 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260131 03:14:31.344442 9922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.344362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:35033 (local address 127.0.0.1:57344)
0131 03:14:31.344406 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.344428 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260131 03:14:31.344545 9920 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.60:35033: Network error: Client connection negotiation failed: client connection to 127.19.84.60:35033: connect: Connection refused (error 111)
I20260131 03:14:31.358129 9997 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } attempt: 1
I20260131 03:14:31.358618 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.358496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:35033 (local address 127.0.0.1:57350)
0131 03:14:31.358558 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.358586 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260131 03:14:31.358968 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.358857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:35033 (local address 127.0.0.1:57354)
0131 03:14:31.358920 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.358948 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260131 03:14:31.359050 9997 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.60:35033: Network error: Client connection negotiation failed: client connection to 127.19.84.60:35033: connect: Connection refused (error 111)
I20260131 03:14:31.375698 9920 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } attempt: 1
I20260131 03:14:31.376245 9922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.376107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:35033 (local address 127.0.0.1:57362)
0131 03:14:31.376175 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.376214 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260131 03:14:31.376632 9922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.376547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:35033 (local address 127.0.0.1:57368)
0131 03:14:31.376592 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.376616 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260131 03:14:31.376751 9920 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.60:35033: Network error: Client connection negotiation failed: client connection to 127.19.84.60:35033: connect: Connection refused (error 111)
W20260131 03:14:31.418591 10006 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260131 03:14:31.418776 10006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:14:31.418792 10006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:14:31.418814 10006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:14:31.420312 10006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:14:31.420356 10006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:14:31.420370 10006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:14:31.420383 10006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:14:31.420395 10006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:14:31.422005 10006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:36877,127.19.84.61:40779,127.19.84.60:35033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.60:35033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.84.60
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:14:31.422242 10006 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:14:31.422447 10006 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:14:31.424873 10011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:31.424889 10014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:31.424962 10012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:31.425148 10006 server_base.cc:1060] running on GCE node
I20260131 03:14:31.425369 10006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:14:31.425598 10006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:14:31.426759 10006 hybrid_clock.cc:648] HybridClock initialized: now 1769829271426732 us; error 36 us; skew 500 ppm
I20260131 03:14:31.427186 9997 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } attempt: 2
I20260131 03:14:31.427600 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.427462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:35033 (local address 127.0.0.1:57374)
0131 03:14:31.427528 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.427564 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260131 03:14:31.428025 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.427921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:35033 (local address 127.0.0.1:57382)
0131 03:14:31.427971 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.427999 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260131 03:14:31.428128 9997 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.60:35033: Network error: Client connection negotiation failed: client connection to 127.19.84.60:35033: connect: Connection refused (error 111)
Jan 31 03:14:31 dist-test-slave-5nnp krb5kdc[9836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829271, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:14:31.429312 10006 init.cc:377] Logged in from keytab as kudu/127.19.84.60@KRBTEST.COM (short username kudu)
I20260131 03:14:31.429876 10006 webserver.cc:492] Webserver started at http://127.19.84.60:42593/ using document root <none> and password file <none>
I20260131 03:14:31.430074 10006 fs_manager.cc:362] Metadata directory not provided
I20260131 03:14:31.430111 10006 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:14:31.430238 10006 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:14:31.431183 10006 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-2/data/instance:
uuid: "b33158bbf4794998bfef8ff22c23b164"
format_stamp: "Formatted at 2026-01-31 03:14:31 on dist-test-slave-5nnp"
server_key: "cbb95125018305ae12bb3d79ddf69674"
server_key_iv: "ca224ec45dc6f9e1ce949faa0a33b555"
server_key_version: "encryptionkey@0"
I20260131 03:14:31.431522 10006 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-2/wal/instance:
uuid: "b33158bbf4794998bfef8ff22c23b164"
format_stamp: "Formatted at 2026-01-31 03:14:31 on dist-test-slave-5nnp"
server_key: "cbb95125018305ae12bb3d79ddf69674"
server_key_iv: "ca224ec45dc6f9e1ce949faa0a33b555"
server_key_version: "encryptionkey@0"
I20260131 03:14:31.432705 10006 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:14:31.433416 10021 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:14:31.433580 10006 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260131 03:14:31.433648 10006 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-2/wal
uuid: "b33158bbf4794998bfef8ff22c23b164"
format_stamp: "Formatted at 2026-01-31 03:14:31 on dist-test-slave-5nnp"
server_key: "cbb95125018305ae12bb3d79ddf69674"
server_key_iv: "ca224ec45dc6f9e1ce949faa0a33b555"
server_key_version: "encryptionkey@0"
I20260131 03:14:31.433696 10006 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:14:31.447605 10006 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:14:31.449263 10006 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:14:31.449364 10006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:14:31.451890 9920 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } attempt: 2
I20260131 03:14:31.452294 9922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.452163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:35033 (local address 127.0.0.1:57390)
0131 03:14:31.452222 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.452264 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260131 03:14:31.452687 9922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.452599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:35033 (local address 127.0.0.1:57400)
0131 03:14:31.452635 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.452667 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:35033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260131 03:14:31.452782 9920 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.60:35033: Network error: Client connection negotiation failed: client connection to 127.19.84.60:35033: connect: Connection refused (error 111)
I20260131 03:14:31.454020 10006 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.60:35033
I20260131 03:14:31.454049 10073 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.60:35033 every 8 connection(s)
I20260131 03:14:31.454355 10006 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-2/data/info.pb
I20260131 03:14:31.454922 10074 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:14:31.456959 10074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36877 } has no permanent_uuid. Determining permanent_uuid...
Jan 31 03:14:31 dist-test-slave-5nnp krb5kdc[9836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829271, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.60@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:14:31.463965 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 10006
I20260131 03:14:31.464092 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1769828849314684-19792-0/minicluster-data/master-2/wal/instance
I20260131 03:14:31.464289 19792 external_mini_cluster.cc:1468] Setting key e1937b0f2ba92f8438911753f7dcbc5e
I20260131 03:14:31.465286 9923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.457688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53654 (local address 127.19.84.62:36877)
0131 03:14:31.457769 (+ 81us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.457772 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.457934 (+ 162us) server_negotiation.cc:408] Connection header received
0131 03:14:31.458308 (+ 374us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.458309 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.458320 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.458375 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.459114 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.459599 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:31.460146 (+ 547us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.460310 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:31.462297 (+ 1987us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:31.462301 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:31.462301 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:31.462309 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:31.463741 (+ 1432us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.464250 (+ 509us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.464251 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.464251 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.464273 (+ 22us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.464551 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.464552 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.464553 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.464685 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:31.464766 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:31.465102 (+ 336us) server_negotiation.cc:300] Negotiation successful
0131 03:14:31.465209 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260131 03:14:31.465462 10076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.457582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:36877 (local address 127.0.0.1:53654)
0131 03:14:31.457807 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.457835 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0131 03:14:31.458211 (+ 376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:14:31.458436 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:14:31.458440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:14:31.458729 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:14:31.459005 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:31.459008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.459702 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:14:31.459703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:14:31.460064 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:31.460065 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.460204 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:31.460769 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:14:31.460787 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:14:31.462211 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:14:31.463832 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:31.463839 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:31.463842 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:31.464168 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:31.464343 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:31.464344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:31.464344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:31.464433 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:31.464779 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:14:31.464783 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:14:31.464970 (+ 187us) client_negotiation.cc:769] Sending connection context
0131 03:14:31.465073 (+ 103us) client_negotiation.cc:240] Negotiation successful
0131 03:14:31.465241 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":101,"threads_started":1}
I20260131 03:14:31.465993 10074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 } has no permanent_uuid. Determining permanent_uuid...
Jan 31 03:14:31 dist-test-slave-5nnp krb5kdc[9836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829271, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.60@KRBTEST.COM for kudu/127.19.84.61@KRBTEST.COM
Jan 31 03:14:31 dist-test-slave-5nnp krb5kdc[9836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.60@KRBTEST.COM
I20260131 03:14:31.472543 10001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.466367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54144 (local address 127.19.84.61:40779)
0131 03:14:31.466437 (+ 70us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.466440 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.466465 (+ 25us) server_negotiation.cc:408] Connection header received
0131 03:14:31.466518 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.466519 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.466529 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.466583 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.466913 (+ 330us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.467552 (+ 639us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:31.468204 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.468379 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:31.469960 (+ 1581us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:31.469965 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:31.469966 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:31.469973 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:31.471348 (+ 1375us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.471819 (+ 471us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.471820 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.471820 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.471844 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.472028 (+ 184us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.472029 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.472030 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.472135 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:31.472189 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:31.472398 (+ 209us) server_negotiation.cc:300] Negotiation successful
0131 03:14:31.472480 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260131 03:14:31.472538 10076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.466296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:40779 (local address 127.0.0.1:54144)
0131 03:14:31.466416 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.466427 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:14:31.466474 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:14:31.466594 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:14:31.466595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:14:31.466702 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:14:31.466840 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:31.466841 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.467622 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:14:31.467623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:14:31.468129 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:31.468131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.468185 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:31.468551 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:14:31.468555 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:14:31.469866 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:14:31.471493 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:31.471494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:31.471495 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:31.471746 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:31.471909 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:31.471910 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:31.471910 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:31.471959 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:31.472217 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:14:31.472218 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:14:31.472265 (+ 47us) client_negotiation.cc:769] Sending connection context
0131 03:14:31.472345 (+ 80us) client_negotiation.cc:240] Negotiation successful
0131 03:14:31.472442 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260131 03:14:31.472977 10074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } has no permanent_uuid. Determining permanent_uuid...
I20260131 03:14:31.474328 10078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.464798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57406 (local address 127.19.84.60:35033)
0131 03:14:31.464977 (+ 179us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.464985 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.465018 (+ 33us) server_negotiation.cc:408] Connection header received
0131 03:14:31.465039 (+ 21us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.465041 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.465061 (+ 20us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.465129 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.466257 (+ 1128us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.467208 (+ 951us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:31.467900 (+ 692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.468067 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:31.470884 (+ 2817us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:31.470892 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:31.470893 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:31.470898 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:31.473016 (+ 2118us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.473572 (+ 556us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.473575 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.473575 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.473618 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.473783 (+ 165us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.473783 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.473784 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.473875 (+ 91us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:31.473981 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:31.474138 (+ 157us) server_negotiation.cc:300] Negotiation successful
0131 03:14:31.474229 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":84,"threads_started":1}
Jan 31 03:14:31 dist-test-slave-5nnp krb5kdc[9836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829271, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.60@KRBTEST.COM for kudu/127.19.84.60@KRBTEST.COM
Jan 31 03:14:31 dist-test-slave-5nnp krb5kdc[9836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:14:31.480643 10076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.473256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:35033 (local address 127.0.0.1:57410)
0131 03:14:31.473308 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.473319 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0131 03:14:31.473358 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:14:31.473617 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:14:31.473618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:14:31.473782 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:14:31.473965 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:31.473967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.474779 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:14:31.474779 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:14:31.475323 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:31.475325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.475415 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:31.475814 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:14:31.475819 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:14:31.477334 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:14:31.479650 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:31.479651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:31.479651 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:31.479834 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:31.480060 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:31.480061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:31.480062 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:31.480090 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:31.480385 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:14:31.480386 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:14:31.480440 (+ 54us) client_negotiation.cc:769] Sending connection context
0131 03:14:31.480499 (+ 59us) client_negotiation.cc:240] Negotiation successful
0131 03:14:31.480576 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260131 03:14:31.480778 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.473315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57410 (local address 127.19.84.60:35033)
0131 03:14:31.473476 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.473478 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.473488 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:14:31.473513 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.473513 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.473521 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.473604 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.474043 (+ 439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.474717 (+ 674us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:31.475437 (+ 720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.475565 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:31.477413 (+ 1848us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:31.477418 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:31.477418 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:31.477423 (+ 5us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:31.479568 (+ 2145us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.479915 (+ 347us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.479916 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.479917 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.479951 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.480200 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.480201 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.480201 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.480284 (+ 83us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:31.480382 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:31.480553 (+ 171us) server_negotiation.cc:300] Negotiation successful
0131 03:14:31.480647 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":114,"threads_started":1}
I20260131 03:14:31.482797 10074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b33158bbf4794998bfef8ff22c23b164: Bootstrap starting.
I20260131 03:14:31.482852 9923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.475121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53660 (local address 127.19.84.62:36877)
0131 03:14:31.475173 (+ 52us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.475175 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.475199 (+ 24us) server_negotiation.cc:408] Connection header received
0131 03:14:31.475220 (+ 21us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.475221 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.475228 (+ 7us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.475274 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.476377 (+ 1103us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.477056 (+ 679us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:31.477689 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.477869 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:31.479853 (+ 1984us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:31.479857 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:31.479858 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:31.479893 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:31.481521 (+ 1628us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.481910 (+ 389us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.481912 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.481912 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.481948 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.482163 (+ 215us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.482164 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.482165 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.482270 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:31.482411 (+ 141us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:31.482670 (+ 259us) server_negotiation.cc:300] Negotiation successful
0131 03:14:31.482756 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260131 03:14:31.483557 10074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b33158bbf4794998bfef8ff22c23b164: Neither blocks nor log segments found. Creating new log.
I20260131 03:14:31.483868 10074 log.cc:826] T 00000000000000000000000000000000 P b33158bbf4794998bfef8ff22c23b164: Log is configured to *not* fsync() on all Append() calls
I20260131 03:14:31.484778 10074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b33158bbf4794998bfef8ff22c23b164: No bootstrap required, opened a new log
I20260131 03:14:31.486016 10074 raft_consensus.cc:359] T 00000000000000000000000000000000 P b33158bbf4794998bfef8ff22c23b164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36877 } } peers { permanent_uuid: "46770edc85f54c77a05592a6b729c63c" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 } } peers { permanent_uuid: "b33158bbf4794998bfef8ff22c23b164" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } }
I20260131 03:14:31.486158 10074 raft_consensus.cc:740] T 00000000000000000000000000000000 P b33158bbf4794998bfef8ff22c23b164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b33158bbf4794998bfef8ff22c23b164, State: Initialized, Role: FOLLOWER
I20260131 03:14:31.486229 10074 consensus_queue.cc:260] T 00000000000000000000000000000000 P b33158bbf4794998bfef8ff22c23b164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36877 } } peers { permanent_uuid: "46770edc85f54c77a05592a6b729c63c" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 } } peers { permanent_uuid: "b33158bbf4794998bfef8ff22c23b164" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } }
I20260131 03:14:31.486531 10081 sys_catalog.cc:455] T 00000000000000000000000000000000 P b33158bbf4794998bfef8ff22c23b164 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36877 } } peers { permanent_uuid: "46770edc85f54c77a05592a6b729c63c" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 } } peers { permanent_uuid: "b33158bbf4794998bfef8ff22c23b164" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } } }
I20260131 03:14:31.486645 10081 sys_catalog.cc:458] T 00000000000000000000000000000000 P b33158bbf4794998bfef8ff22c23b164 [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:14:31.486548 10074 sys_catalog.cc:565] T 00000000000000000000000000000000 P b33158bbf4794998bfef8ff22c23b164 [sys.catalog]: configured and running, proceeding with master startup.
W20260131 03:14:31.488723 10092 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b33158bbf4794998bfef8ff22c23b164: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:14:31.488788 10092 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:14:31.489223 10006 master_runner.cc:186] Error getting master registration for 127.19.84.62:36877: OK, instance_id { permanent_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" instance_seqno: 1769829271110744 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260131 03:14:31.490731 10006 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jan 31 03:14:31 dist-test-slave-5nnp krb5kdc[9836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.61@KRBTEST.COM
I20260131 03:14:31.496376 10001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.483706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.19.84.61:40779)
0131 03:14:31.483778 (+ 72us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.483780 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.483792 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:14:31.483816 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.483817 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.483827 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.483896 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.484629 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.485278 (+ 649us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:31.485991 (+ 713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.486213 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:31.493447 (+ 7234us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:31.493455 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:31.493455 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:31.493464 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:31.495108 (+ 1644us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.495653 (+ 545us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.495655 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.495655 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.495686 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.495854 (+ 168us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.495855 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.495855 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.495983 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:31.496143 (+ 160us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:31.496181 (+ 38us) server_negotiation.cc:300] Negotiation successful
0131 03:14:31.496275 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260131 03:14:31.498039 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:14:31.498862 9894 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53660
I20260131 03:14:31.519300 9997 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } attempt: 3
Jan 31 03:14:31 dist-test-slave-5nnp krb5kdc[9836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829271, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.61@KRBTEST.COM for kudu/127.19.84.60@KRBTEST.COM
I20260131 03:14:31.526091 9999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.519778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:35033 (local address 127.0.0.1:57416)
0131 03:14:31.519826 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.519839 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:14:31.519891 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:14:31.520087 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:14:31.520088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:14:31.520275 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:14:31.520410 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:31.520412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.521100 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:14:31.521101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:14:31.521467 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:31.521469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.521561 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:31.522132 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:14:31.522138 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:14:31.523481 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:14:31.525149 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:31.525150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:31.525151 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:31.525312 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:31.525482 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:31.525483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:31.525483 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:31.525513 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:31.525751 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:14:31.525752 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:14:31.525798 (+ 46us) client_negotiation.cc:769] Sending connection context
0131 03:14:31.525887 (+ 89us) client_negotiation.cc:240] Negotiation successful
0131 03:14:31.525991 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260131 03:14:31.526091 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.519890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57416 (local address 127.19.84.60:35033)
0131 03:14:31.519957 (+ 67us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.519959 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.519967 (+ 8us) server_negotiation.cc:408] Connection header received
0131 03:14:31.520000 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.520001 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.520012 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.520079 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.520508 (+ 429us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.521006 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:31.521554 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.521762 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:31.523561 (+ 1799us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:31.523565 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:31.523565 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:31.523573 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:31.525049 (+ 1476us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.525382 (+ 333us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.525383 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.525384 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.525411 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.525567 (+ 156us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.525568 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.525569 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.525687 (+ 118us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:31.525762 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:31.525917 (+ 155us) server_negotiation.cc:300] Negotiation successful
0131 03:14:31.525999 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260131 03:14:31.527662 9997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46770edc85f54c77a05592a6b729c63c: Bootstrap starting.
I20260131 03:14:31.528396 9997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46770edc85f54c77a05592a6b729c63c: Neither blocks nor log segments found. Creating new log.
I20260131 03:14:31.528613 9997 log.cc:826] T 00000000000000000000000000000000 P 46770edc85f54c77a05592a6b729c63c: Log is configured to *not* fsync() on all Append() calls
I20260131 03:14:31.529307 9997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46770edc85f54c77a05592a6b729c63c: No bootstrap required, opened a new log
I20260131 03:14:31.530383 9997 raft_consensus.cc:359] T 00000000000000000000000000000000 P 46770edc85f54c77a05592a6b729c63c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36877 } } peers { permanent_uuid: "46770edc85f54c77a05592a6b729c63c" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 } } peers { permanent_uuid: "b33158bbf4794998bfef8ff22c23b164" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } }
I20260131 03:14:31.530478 9997 raft_consensus.cc:740] T 00000000000000000000000000000000 P 46770edc85f54c77a05592a6b729c63c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46770edc85f54c77a05592a6b729c63c, State: Initialized, Role: FOLLOWER
I20260131 03:14:31.530520 9997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46770edc85f54c77a05592a6b729c63c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36877 } } peers { permanent_uuid: "46770edc85f54c77a05592a6b729c63c" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 } } peers { permanent_uuid: "b33158bbf4794998bfef8ff22c23b164" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } }
I20260131 03:14:31.530740 10095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46770edc85f54c77a05592a6b729c63c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36877 } } peers { permanent_uuid: "46770edc85f54c77a05592a6b729c63c" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 } } peers { permanent_uuid: "b33158bbf4794998bfef8ff22c23b164" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } } }
I20260131 03:14:31.530782 9997 sys_catalog.cc:565] T 00000000000000000000000000000000 P 46770edc85f54c77a05592a6b729c63c [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:14:31.530812 10095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46770edc85f54c77a05592a6b729c63c [sys.catalog]: This master's current role is: FOLLOWER
W20260131 03:14:31.532821 10106 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 46770edc85f54c77a05592a6b729c63c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:14:31.532881 10106 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260131 03:14:31.534572 9950 tablet.cc:2378] T 00000000000000000000000000000000 P 46770edc85f54c77a05592a6b729c63c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:14:31.533010 9929 master_runner.cc:186] Error getting master registration for 127.19.84.62:36877: OK, instance_id { permanent_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" instance_seqno: 1769829271110744 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260131 03:14:31.535081 9929 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260131 03:14:31.551954 9920 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } attempt: 3
Jan 31 03:14:31 dist-test-slave-5nnp krb5kdc[9836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829271, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for kudu/127.19.84.60@KRBTEST.COM
I20260131 03:14:31.559142 9922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.552377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:35033 (local address 127.0.0.1:57424)
0131 03:14:31.552444 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:31.552456 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0131 03:14:31.552501 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:14:31.552682 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:14:31.552683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:14:31.552897 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:14:31.553105 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:31.553108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.553985 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:14:31.553986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:14:31.554338 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:31.554340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.554426 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:31.554985 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:14:31.554990 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:14:31.556576 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:14:31.558207 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:31.558208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:31.558208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:31.558394 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:31.558571 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:31.558572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:31.558573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:31.558603 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:31.558813 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:14:31.558814 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:14:31.558862 (+ 48us) client_negotiation.cc:769] Sending connection context
0131 03:14:31.558932 (+ 70us) client_negotiation.cc:240] Negotiation successful
0131 03:14:31.559032 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260131 03:14:31.559548 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.552432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57424 (local address 127.19.84.60:35033)
0131 03:14:31.552569 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.552571 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.552578 (+ 7us) server_negotiation.cc:408] Connection header received
0131 03:14:31.552601 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.552602 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.552610 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.552674 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.553209 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.553907 (+ 698us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:31.554421 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.554614 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:31.556664 (+ 2050us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:31.556668 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:31.556668 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:31.556675 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:31.558073 (+ 1398us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.558464 (+ 391us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.558465 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.558465 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.558495 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.558651 (+ 156us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.558651 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.558652 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.558749 (+ 97us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:31.558825 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:31.559347 (+ 522us) server_negotiation.cc:300] Negotiation successful
0131 03:14:31.559452 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260131 03:14:31.561017 9920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1: Bootstrap starting.
I20260131 03:14:31.561725 9920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1: Neither blocks nor log segments found. Creating new log.
I20260131 03:14:31.562000 9920 log.cc:826] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1: Log is configured to *not* fsync() on all Append() calls
I20260131 03:14:31.562781 9920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1: No bootstrap required, opened a new log
I20260131 03:14:31.563839 9920 raft_consensus.cc:359] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36877 } } peers { permanent_uuid: "46770edc85f54c77a05592a6b729c63c" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 } } peers { permanent_uuid: "b33158bbf4794998bfef8ff22c23b164" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } }
I20260131 03:14:31.563963 9920 raft_consensus.cc:740] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd7d457eb5dc43e897a21b81dfcb01f1, State: Initialized, Role: FOLLOWER
I20260131 03:14:31.564028 9920 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36877 } } peers { permanent_uuid: "46770edc85f54c77a05592a6b729c63c" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 } } peers { permanent_uuid: "b33158bbf4794998bfef8ff22c23b164" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } }
I20260131 03:14:31.564293 9920 sys_catalog.cc:565] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:14:31.564337 9894 raft_consensus.cc:493] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260131 03:14:31.564389 9894 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:14:31.565281 9894 raft_consensus.cc:515] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36877 } } peers { permanent_uuid: "46770edc85f54c77a05592a6b729c63c" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 } } peers { permanent_uuid: "b33158bbf4794998bfef8ff22c23b164" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } }
I20260131 03:14:31.565503 9894 leader_election.cc:290] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1 [CANDIDATE]: Term 1 election: Requested vote from peers 46770edc85f54c77a05592a6b729c63c (127.19.84.61:40779), b33158bbf4794998bfef8ff22c23b164 (127.19.84.60:35033)
I20260131 03:14:31.565973 9971 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "46770edc85f54c77a05592a6b729c63c"
I20260131 03:14:31.566087 9971 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 46770edc85f54c77a05592a6b729c63c [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:14:31.566059 10048 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b33158bbf4794998bfef8ff22c23b164"
I20260131 03:14:31.566144 10048 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b33158bbf4794998bfef8ff22c23b164 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:14:31.566972 9971 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 46770edc85f54c77a05592a6b729c63c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd7d457eb5dc43e897a21b81dfcb01f1 in term 1.
I20260131 03:14:31.566968 10048 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b33158bbf4794998bfef8ff22c23b164 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd7d457eb5dc43e897a21b81dfcb01f1 in term 1.
I20260131 03:14:31.567510 9868 leader_election.cc:304] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b33158bbf4794998bfef8ff22c23b164, fd7d457eb5dc43e897a21b81dfcb01f1; no voters:
I20260131 03:14:31.568816 10109 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36877 } } peers { permanent_uuid: "46770edc85f54c77a05592a6b729c63c" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 } } peers { permanent_uuid: "b33158bbf4794998bfef8ff22c23b164" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } } }
I20260131 03:14:31.568890 10109 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:14:31.568931 10109 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:14:31.568986 10109 raft_consensus.cc:697] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1 [term 1 LEADER]: Becoming Leader. State: Replica: fd7d457eb5dc43e897a21b81dfcb01f1, State: Running, Role: LEADER
I20260131 03:14:31.569060 10109 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36877 } } peers { permanent_uuid: "46770edc85f54c77a05592a6b729c63c" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 } } peers { permanent_uuid: "b33158bbf4794998bfef8ff22c23b164" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } }
I20260131 03:14:31.569440 10109 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd7d457eb5dc43e897a21b81dfcb01f1. Latest consensus state: current_term: 1 leader_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36877 } } peers { permanent_uuid: "46770edc85f54c77a05592a6b729c63c" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 } } peers { permanent_uuid: "b33158bbf4794998bfef8ff22c23b164" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } } }
I20260131 03:14:31.569514 10109 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1 [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:31.569669 9852 master_runner.cc:263] No leader master found from master fd7d457eb5dc43e897a21b81dfcb01f1
I20260131 03:14:31.569711 9852 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260131 03:14:31.574877 10120 catalog_manager.cc:1568] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:14:31.576031 10120 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:14:31.580507 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.569957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57440 (local address 127.19.84.60:35033)
0131 03:14:31.569994 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.569996 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.570584 (+ 588us) server_negotiation.cc:408] Connection header received
0131 03:14:31.570682 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.570683 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.570692 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.570723 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.571595 (+ 872us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.572251 (+ 656us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:31.572959 (+ 708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.573145 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:31.576226 (+ 3081us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:31.576229 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:31.576230 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:31.576237 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:31.579251 (+ 3014us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.579660 (+ 409us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.579661 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.579662 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.579687 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.579863 (+ 176us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.579864 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.579864 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.579985 (+ 121us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:31.580104 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:31.580319 (+ 215us) server_negotiation.cc:300] Negotiation successful
0131 03:14:31.580408 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260131 03:14:31.579274 10127 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:14:31.582669 10127 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:14:31.583314 10001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.570316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54168 (local address 127.19.84.61:40779)
0131 03:14:31.570358 (+ 42us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.570360 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.573574 (+ 3214us) server_negotiation.cc:408] Connection header received
0131 03:14:31.573682 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.573683 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.573693 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.573727 (+ 34us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.574613 (+ 886us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.575257 (+ 644us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:31.576477 (+ 1220us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.576672 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:31.580407 (+ 3735us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:31.580411 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:31.580412 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:31.580419 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:31.582154 (+ 1735us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.582521 (+ 367us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.582523 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.582523 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.582548 (+ 25us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.582706 (+ 158us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.582707 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.582708 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.582819 (+ 111us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:31.582907 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:31.583135 (+ 228us) server_negotiation.cc:300] Negotiation successful
0131 03:14:31.583223 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260131 03:14:31.583313 9923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:31.568591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53668 (local address 127.19.84.62:36877)
0131 03:14:31.568628 (+ 37us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.568631 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.569878 (+ 1247us) server_negotiation.cc:408] Connection header received
0131 03:14:31.569999 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.570001 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.570011 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.570042 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.571120 (+ 1078us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.571829 (+ 709us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:31.572548 (+ 719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:31.572708 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:31.576058 (+ 3350us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:31.576062 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:31.576063 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:31.576070 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:31.577828 (+ 1758us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.581733 (+ 3905us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.581735 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.581735 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.581761 (+ 26us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:31.582792 (+ 1031us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:31.582793 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:31.582794 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:31.582895 (+ 101us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:31.582960 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:31.583147 (+ 187us) server_negotiation.cc:300] Negotiation successful
0131 03:14:31.583233 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260131 03:14:31.585968 9971 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 46770edc85f54c77a05592a6b729c63c [term 1 FOLLOWER]: Refusing update from remote peer fd7d457eb5dc43e897a21b81dfcb01f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:14:31.586077 10048 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b33158bbf4794998bfef8ff22c23b164 [term 1 FOLLOWER]: Refusing update from remote peer fd7d457eb5dc43e897a21b81dfcb01f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:14:31.586259 10109 consensus_queue.cc:1048] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46770edc85f54c77a05592a6b729c63c" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:14:31.586385 10121 consensus_queue.cc:1048] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b33158bbf4794998bfef8ff22c23b164" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:14:31.587508 10081 sys_catalog.cc:455] T 00000000000000000000000000000000 P b33158bbf4794998bfef8ff22c23b164 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd7d457eb5dc43e897a21b81dfcb01f1. Latest consensus state: current_term: 1 leader_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36877 } } peers { permanent_uuid: "46770edc85f54c77a05592a6b729c63c" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 } } peers { permanent_uuid: "b33158bbf4794998bfef8ff22c23b164" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } } }
I20260131 03:14:31.587601 10081 sys_catalog.cc:458] T 00000000000000000000000000000000 P b33158bbf4794998bfef8ff22c23b164 [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:14:31.587568 10095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46770edc85f54c77a05592a6b729c63c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd7d457eb5dc43e897a21b81dfcb01f1. Latest consensus state: current_term: 1 leader_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36877 } } peers { permanent_uuid: "46770edc85f54c77a05592a6b729c63c" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 } } peers { permanent_uuid: "b33158bbf4794998bfef8ff22c23b164" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } } }
I20260131 03:14:31.587642 10095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46770edc85f54c77a05592a6b729c63c [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:14:31.588546 10109 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36877 } } peers { permanent_uuid: "46770edc85f54c77a05592a6b729c63c" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 } } peers { permanent_uuid: "b33158bbf4794998bfef8ff22c23b164" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } } }
I20260131 03:14:31.588620 10109 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1 [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:31.588951 10095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46770edc85f54c77a05592a6b729c63c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36877 } } peers { permanent_uuid: "46770edc85f54c77a05592a6b729c63c" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 } } peers { permanent_uuid: "b33158bbf4794998bfef8ff22c23b164" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } } }
I20260131 03:14:31.589020 10095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46770edc85f54c77a05592a6b729c63c [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:14:31.590147 10132 mvcc.cc:204] Tried to move back new op lower bound from 7249220696412975104 to 7249220696347353088. Current Snapshot: MvccSnapshot[applied={T|T < 7249220696412975104}]
I20260131 03:14:31.590634 10134 mvcc.cc:204] Tried to move back new op lower bound from 7249220696412975104 to 7249220696347353088. Current Snapshot: MvccSnapshot[applied={T|T < 7249220696412975104}]
I20260131 03:14:31.591091 10127 catalog_manager.cc:1357] Generated new cluster ID: ccd8a7e965e349e2b8ea574e5dd2663a
I20260131 03:14:31.591168 10127 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:14:31.600379 10121 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36877 } } peers { permanent_uuid: "46770edc85f54c77a05592a6b729c63c" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 } } peers { permanent_uuid: "b33158bbf4794998bfef8ff22c23b164" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } } }
I20260131 03:14:31.600477 10121 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1 [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:31.600884 10081 sys_catalog.cc:455] T 00000000000000000000000000000000 P b33158bbf4794998bfef8ff22c23b164 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d457eb5dc43e897a21b81dfcb01f1" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36877 } } peers { permanent_uuid: "46770edc85f54c77a05592a6b729c63c" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 40779 } } peers { permanent_uuid: "b33158bbf4794998bfef8ff22c23b164" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 35033 } } }
I20260131 03:14:31.600981 10081 sys_catalog.cc:458] T 00000000000000000000000000000000 P b33158bbf4794998bfef8ff22c23b164 [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:14:31.601068 10133 mvcc.cc:204] Tried to move back new op lower bound from 7249220696412975104 to 7249220696347353088. Current Snapshot: MvccSnapshot[applied={T|T < 7249220696412975104}]
I20260131 03:14:31.610874 10127 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:14:31.611658 10127 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:14:31.628669 10127 catalog_manager.cc:6027] T 00000000000000000000000000000000 P fd7d457eb5dc43e897a21b81dfcb01f1: Generated new TSK 0
I20260131 03:14:31.628911 10127 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260131 03:14:31.650866 9923 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:31.649496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53676 (local address 127.19.84.62:36877)
0131 03:14:31.649542 (+ 46us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.649544 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.650196 (+ 652us) server_negotiation.cc:408] Connection header received
0131 03:14:31.650312 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.650313 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.650324 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.650355 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.650798 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53676: BlockingRecv error: recv got EOF from 127.0.0.1:53676 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260131 03:14:31.651706 10001 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:31.649911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54180 (local address 127.19.84.61:40779)
0131 03:14:31.649966 (+ 55us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.649968 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.651185 (+ 1217us) server_negotiation.cc:408] Connection header received
0131 03:14:31.651285 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.651286 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.651297 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.651326 (+ 29us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.651651 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54180: BlockingRecv error: recv got EOF from 127.0.0.1:54180 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260131 03:14:31.652745 10079 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:31.652117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57454 (local address 127.19.84.60:35033)
0131 03:14:31.652157 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.652159 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.652266 (+ 107us) server_negotiation.cc:408] Connection header received
0131 03:14:31.652356 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.652357 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.652367 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.652396 (+ 29us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.652694 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57454: BlockingRecv error: recv got EOF from 127.0.0.1:57454 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260131 03:14:31.655016 10001 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:31.654307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54194 (local address 127.19.84.61:40779)
0131 03:14:31.654342 (+ 35us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.654343 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.654513 (+ 170us) server_negotiation.cc:408] Connection header received
0131 03:14:31.654616 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.654617 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.654631 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.654659 (+ 28us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.654971 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54194: BlockingRecv error: recv got EOF from 127.0.0.1:54194 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260131 03:14:31.658070 10001 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:31.657359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54204 (local address 127.19.84.61:40779)
0131 03:14:31.657390 (+ 31us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.657392 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.657600 (+ 208us) server_negotiation.cc:408] Connection header received
0131 03:14:31.657693 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.657693 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.657702 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.657730 (+ 28us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.658030 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54204: BlockingRecv error: recv got EOF from 127.0.0.1:54204 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260131 03:14:31.661934 10001 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:31.661250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54206 (local address 127.19.84.61:40779)
0131 03:14:31.661283 (+ 33us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.661284 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.661485 (+ 201us) server_negotiation.cc:408] Connection header received
0131 03:14:31.661576 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.661577 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.661586 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.661613 (+ 27us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.661893 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54206: BlockingRecv error: recv got EOF from 127.0.0.1:54206 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260131 03:14:31.668112 10001 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:31.667322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54210 (local address 127.19.84.61:40779)
0131 03:14:31.667355 (+ 33us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.667357 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.667625 (+ 268us) server_negotiation.cc:408] Connection header received
0131 03:14:31.667720 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.667721 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.667731 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.667759 (+ 28us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.668070 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54210: BlockingRecv error: recv got EOF from 127.0.0.1:54210 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260131 03:14:31.678251 10001 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:31.677420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.19.84.61:40779)
0131 03:14:31.677460 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.677462 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.677692 (+ 230us) server_negotiation.cc:408] Connection header received
0131 03:14:31.677792 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.677793 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.677804 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.677833 (+ 29us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.678196 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54214: BlockingRecv error: recv got EOF from 127.0.0.1:54214 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260131 03:14:31.696870 10001 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:31.695903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54224 (local address 127.19.84.61:40779)
0131 03:14:31.695959 (+ 56us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.695962 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.696257 (+ 295us) server_negotiation.cc:408] Connection header received
0131 03:14:31.696366 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.696368 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.696379 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.696410 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.696813 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54224: BlockingRecv error: recv got EOF from 127.0.0.1:54224 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260131 03:14:31.731359 10001 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:31.730549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54234 (local address 127.19.84.61:40779)
0131 03:14:31.730611 (+ 62us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.730614 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.730743 (+ 129us) server_negotiation.cc:408] Connection header received
0131 03:14:31.730808 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.730809 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.730826 (+ 17us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.730907 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.731296 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54234: BlockingRecv error: recv got EOF from 127.0.0.1:54234 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260131 03:14:31.797477 10001 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:31.796689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54242 (local address 127.19.84.61:40779)
0131 03:14:31.796768 (+ 79us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.796770 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.796874 (+ 104us) server_negotiation.cc:408] Connection header received
0131 03:14:31.796992 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.796994 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.797006 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.797079 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.797429 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54242: BlockingRecv error: recv got EOF from 127.0.0.1:54242 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260131 03:14:31.927621 10001 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:31.926916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54258 (local address 127.19.84.61:40779)
0131 03:14:31.927001 (+ 85us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:31.927003 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:31.927028 (+ 25us) server_negotiation.cc:408] Connection header received
0131 03:14:31.927096 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:31.927097 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:31.927107 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:31.927169 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:31.927550 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54258: BlockingRecv error: recv got EOF from 127.0.0.1:54258 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260131 03:14:32.185495 10001 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:32.184882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54272 (local address 127.19.84.61:40779)
0131 03:14:32.184967 (+ 85us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:32.184969 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:32.184979 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:14:32.185012 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:32.185013 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:32.185023 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:32.185095 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:32.185445 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54272: BlockingRecv error: recv got EOF from 127.0.0.1:54272 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260131 03:14:32.489284 10092 catalog_manager.cc:1269] Loaded cluster ID: ccd8a7e965e349e2b8ea574e5dd2663a
I20260131 03:14:32.489382 10092 catalog_manager.cc:1562] T 00000000000000000000000000000000 P b33158bbf4794998bfef8ff22c23b164: loading cluster ID for follower catalog manager: success
I20260131 03:14:32.490898 10092 catalog_manager.cc:1584] T 00000000000000000000000000000000 P b33158bbf4794998bfef8ff22c23b164: acquiring CA information for follower catalog manager: success
I20260131 03:14:32.491240 10092 catalog_manager.cc:1612] T 00000000000000000000000000000000 P b33158bbf4794998bfef8ff22c23b164: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260131 03:14:32.533295 10106 catalog_manager.cc:1269] Loaded cluster ID: ccd8a7e965e349e2b8ea574e5dd2663a
I20260131 03:14:32.533398 10106 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 46770edc85f54c77a05592a6b729c63c: loading cluster ID for follower catalog manager: success
I20260131 03:14:32.534264 10106 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 46770edc85f54c77a05592a6b729c63c: acquiring CA information for follower catalog manager: success
I20260131 03:14:32.534561 10106 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 46770edc85f54c77a05592a6b729c63c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260131 03:14:32.701113 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:32.698812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54288 (local address 127.19.84.61:40779)
0131 03:14:32.698962 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:32.698964 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:32.698977 (+ 13us) server_negotiation.cc:408] Connection header received
0131 03:14:32.699068 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:32.699069 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:32.699109 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:32.699190 (+ 81us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:14:32.699482 (+ 292us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:32.699987 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:32.700539 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:32.700648 (+ 109us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:32.700689 (+ 41us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:14:32.700781 (+ 92us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:14:32.700883 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:32.700990 (+ 107us) server_negotiation.cc:300] Negotiation successful
0131 03:14:32.701016 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":78,"threads_started":1}
I20260131 03:14:32.705269 10210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:32.703004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57458 (local address 127.19.84.60:35033)
0131 03:14:32.703088 (+ 84us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:32.703090 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:32.703097 (+ 7us) server_negotiation.cc:408] Connection header received
0131 03:14:32.703156 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:32.703157 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:32.703194 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:32.703251 (+ 57us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:14:32.703554 (+ 303us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:32.704024 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:32.704748 (+ 724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:32.704856 (+ 108us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:32.704889 (+ 33us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:14:32.704976 (+ 87us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:14:32.705037 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:32.705110 (+ 73us) server_negotiation.cc:300] Negotiation successful
0131 03:14:32.705135 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69,"thread_start_us":44,"threads_started":1}
I20260131 03:14:32.709415 10210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:32.707187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57474 (local address 127.19.84.60:35033)
0131 03:14:32.707235 (+ 48us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:32.707237 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:32.707246 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:14:32.707267 (+ 21us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:32.707268 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:32.707282 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:32.707333 (+ 51us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:14:32.707674 (+ 341us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:32.708174 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:32.708843 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:32.709012 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:32.709031 (+ 19us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:14:32.709056 (+ 25us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:14:32.709141 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:32.709301 (+ 160us) server_negotiation.cc:300] Negotiation successful
0131 03:14:32.709328 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260131 03:14:32.709777 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:32.707150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54302 (local address 127.19.84.61:40779)
0131 03:14:32.707188 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:32.707190 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:32.707332 (+ 142us) server_negotiation.cc:408] Connection header received
0131 03:14:32.707422 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:32.707423 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:32.707435 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:32.707501 (+ 66us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:14:32.707759 (+ 258us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:32.708460 (+ 701us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:32.709101 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:32.709255 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:32.709293 (+ 38us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:14:32.709326 (+ 33us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:14:32.709381 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:32.709676 (+ 295us) server_negotiation.cc:300] Negotiation successful
0131 03:14:32.709706 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260131 03:14:32.710060 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:32.707004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53682 (local address 127.19.84.62:36877)
0131 03:14:32.707389 (+ 385us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:32.707392 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:32.707399 (+ 7us) server_negotiation.cc:408] Connection header received
0131 03:14:32.707448 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:32.707449 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:32.707478 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:32.707840 (+ 362us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:14:32.707886 (+ 46us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:32.708615 (+ 729us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:32.709411 (+ 796us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:32.709602 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:32.709659 (+ 57us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:14:32.709754 (+ 95us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:14:32.709820 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:32.709967 (+ 147us) server_negotiation.cc:300] Negotiation successful
0131 03:14:32.709994 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":69,"threads_started":1}
I20260131 03:14:32.711526 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 9852
I20260131 03:14:32.716571 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 9929
I20260131 03:14:32.721319 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 10006
2026-01-31T03:14:32Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3893 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 31 03:14:32 dist-test-slave-5nnp krb5kdc[10221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 31 03:14:32 dist-test-slave-5nnp krb5kdc[10221](info): set up 2 sockets
Jan 31 03:14:32 dist-test-slave-5nnp krb5kdc[10221](info): commencing operation
krb5kdc: starting...
W20260131 03:14:34.740401 19792 mini_kdc.cc:121] Time spent starting KDC: real 1.999s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/test-user.keytab.
Jan 31 03:14:34 dist-test-slave-5nnp krb5kdc[10221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-31T03:14:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-31T03:14:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:14:34.873822 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:36721
--webserver_interface=127.19.84.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:36721,127.19.84.61:36965,127.19.84.60:39859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260131 03:14:34.954059 10237 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260131 03:14:34.954313 10237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:14:34.954344 10237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:14:34.954393 10237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:14:34.955924 10237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:14:34.956005 10237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:14:34.956030 10237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:14:34.956050 10237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:14:34.956081 10237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:14:34.957724 10237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:36721,127.19.84.61:36965,127.19.84.60:39859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.62:36721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.84.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:14:34.958002 10237 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:14:34.958204 10237 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:14:34.960808 10243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:34.960764 10245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:34.960764 10242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:34.960920 10237 server_base.cc:1060] running on GCE node
I20260131 03:14:34.961334 10237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:14:34.961608 10237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:14:34.962769 10237 hybrid_clock.cc:648] HybridClock initialized: now 1769829274962745 us; error 36 us; skew 500 ppm
Jan 31 03:14:34 dist-test-slave-5nnp krb5kdc[10221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829274, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:14:34.965167 10237 init.cc:377] Logged in from keytab as kudu/127.19.84.62@KRBTEST.COM (short username kudu)
I20260131 03:14:34.965770 10237 webserver.cc:492] Webserver started at http://127.19.84.62:34763/ using document root <none> and password file <none>
I20260131 03:14:34.965978 10237 fs_manager.cc:362] Metadata directory not provided
I20260131 03:14:34.966043 10237 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:14:34.966180 10237 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:14:34.967175 10237 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-0/data/instance:
uuid: "3601bb0cbc3b480bb69a6906ba48d069"
format_stamp: "Formatted at 2026-01-31 03:14:34 on dist-test-slave-5nnp"
server_key: "c0d950e9f83d70fb6abf368ab6441075"
server_key_iv: "41269be74bdb82f5e64b92cafa4fd7af"
server_key_version: "encryptionkey@0"
I20260131 03:14:34.967581 10237 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-0/wal/instance:
uuid: "3601bb0cbc3b480bb69a6906ba48d069"
format_stamp: "Formatted at 2026-01-31 03:14:34 on dist-test-slave-5nnp"
server_key: "c0d950e9f83d70fb6abf368ab6441075"
server_key_iv: "41269be74bdb82f5e64b92cafa4fd7af"
server_key_version: "encryptionkey@0"
I20260131 03:14:34.968842 10237 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:14:34.969594 10252 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:14:34.969811 10237 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260131 03:14:34.969904 10237 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-0/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-0/wal
uuid: "3601bb0cbc3b480bb69a6906ba48d069"
format_stamp: "Formatted at 2026-01-31 03:14:34 on dist-test-slave-5nnp"
server_key: "c0d950e9f83d70fb6abf368ab6441075"
server_key_iv: "41269be74bdb82f5e64b92cafa4fd7af"
server_key_version: "encryptionkey@0"
I20260131 03:14:34.970008 10237 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:14:34.989317 10237 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:14:34.991212 10237 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:14:34.991426 10237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:14:34.996037 10237 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.62:36721
I20260131 03:14:34.996090 10304 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.62:36721 every 8 connection(s)
I20260131 03:14:34.996443 10237 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-0/data/info.pb
I20260131 03:14:34.997135 10305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:14:34.998445 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 10237
I20260131 03:14:34.998565 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-0/wal/instance
I20260131 03:14:34.998724 19792 external_mini_cluster.cc:1468] Setting key eaf37ac3d2175ad140951ca09c6e3a5f
I20260131 03:14:34.999827 10305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36721 } has no permanent_uuid. Determining permanent_uuid...
Jan 31 03:14:35 dist-test-slave-5nnp krb5kdc[10221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829274, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.84.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.61@KRBTEST.COM" created.
I20260131 03:14:35.014218 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.000697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:36721 (local address 127.0.0.1:37396)
0131 03:14:35.003525 (+ 2828us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.003569 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0131 03:14:35.004054 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:14:35.004438 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:14:35.004443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:14:35.004863 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:14:35.005278 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:35.005282 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.006404 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:14:35.006405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:14:35.006752 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:35.006753 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.006924 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:35.008913 (+ 1989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:14:35.008934 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:14:35.010660 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:14:35.012752 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:35.012758 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:35.012761 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:35.012931 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:35.013145 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:35.013146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:35.013147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:35.013209 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:35.013625 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:14:35.013630 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:14:35.013694 (+ 64us) client_negotiation.cc:769] Sending connection context
0131 03:14:35.013804 (+ 110us) client_negotiation.cc:240] Negotiation successful
0131 03:14:35.014029 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2780,"mutex_wait_us":103,"thread_start_us":72,"threads_started":1}
I20260131 03:14:35.014221 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.003456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37396 (local address 127.19.84.62:36721)
0131 03:14:35.004229 (+ 773us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.004248 (+ 19us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.004262 (+ 14us) server_negotiation.cc:408] Connection header received
0131 03:14:35.004328 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.004330 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.004349 (+ 19us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.005370 (+ 1021us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.005418 (+ 48us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.006317 (+ 899us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:35.006901 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.007546 (+ 645us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:35.010765 (+ 3219us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:35.010776 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:35.010776 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:35.010784 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:35.012590 (+ 1806us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.013026 (+ 436us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.013029 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.013029 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.013076 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.013281 (+ 205us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.013282 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.013282 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.013495 (+ 213us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:35.013657 (+ 162us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:35.013824 (+ 167us) server_negotiation.cc:300] Negotiation successful
0131 03:14:35.013967 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":106,"threads_started":1}
I20260131 03:14:35.015218 10305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 } has no permanent_uuid. Determining permanent_uuid...
I20260131 03:14:35.015597 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.015477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:36965 (local address 127.0.0.1:33494)
0131 03:14:35.015527 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.015571 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.61:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260131 03:14:35.015744 10253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.84.61:36965: connect: Connection refused (error 111)
I20260131 03:14:35.016216 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.016108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:36965 (local address 127.0.0.1:33502)
0131 03:14:35.016156 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.016187 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.61:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260131 03:14:35.016407 10305 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.61:36965: Network error: Client connection negotiation failed: client connection to 127.19.84.61:36965: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:14:35.056556 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.61:36965
--webserver_interface=127.19.84.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:36721,127.19.84.61:36965,127.19.84.60:39859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260131 03:14:35.065702 10305 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 } attempt: 1
I20260131 03:14:35.066219 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.066081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:36965 (local address 127.0.0.1:33516)
0131 03:14:35.066150 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.066189 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.61:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260131 03:14:35.066613 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.066533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:36965 (local address 127.0.0.1:33532)
0131 03:14:35.066573 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.066596 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.61:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260131 03:14:35.066722 10305 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.61:36965: Network error: Client connection negotiation failed: client connection to 127.19.84.61:36965: connect: Connection refused (error 111)
I20260131 03:14:35.134894 10305 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 } attempt: 2
I20260131 03:14:35.135500 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.135320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:36965 (local address 127.0.0.1:33534)
0131 03:14:35.135433 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.135473 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.61:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260131 03:14:35.135898 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.135804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:36965 (local address 127.0.0.1:33546)
0131 03:14:35.135848 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.135872 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.61:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260131 03:14:35.136029 10305 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.61:36965: Network error: Client connection negotiation failed: client connection to 127.19.84.61:36965: connect: Connection refused (error 111)
W20260131 03:14:35.137450 10314 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260131 03:14:35.137651 10314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:14:35.137705 10314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:14:35.137748 10314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:14:35.139207 10314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:14:35.139283 10314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:14:35.139309 10314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:14:35.139338 10314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:14:35.139360 10314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:14:35.140937 10314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:36721,127.19.84.61:36965,127.19.84.60:39859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.61:36965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.84.61
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:14:35.141202 10314 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:14:35.141402 10314 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:14:35.143766 10322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:35.143765 10320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:35.143769 10319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:35.144065 10314 server_base.cc:1060] running on GCE node
I20260131 03:14:35.144299 10314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:14:35.144533 10314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:14:35.145704 10314 hybrid_clock.cc:648] HybridClock initialized: now 1769829275145688 us; error 23 us; skew 500 ppm
Jan 31 03:14:35 dist-test-slave-5nnp krb5kdc[10221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829275, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:14:35.148048 10314 init.cc:377] Logged in from keytab as kudu/127.19.84.61@KRBTEST.COM (short username kudu)
I20260131 03:14:35.148586 10314 webserver.cc:492] Webserver started at http://127.19.84.61:42149/ using document root <none> and password file <none>
I20260131 03:14:35.148742 10314 fs_manager.cc:362] Metadata directory not provided
I20260131 03:14:35.148773 10314 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:14:35.148852 10314 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:14:35.149744 10314 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-1/data/instance:
uuid: "13f76f18a15b4d2e82e79cf574f4f34e"
format_stamp: "Formatted at 2026-01-31 03:14:35 on dist-test-slave-5nnp"
server_key: "5a0c5275f9b1e5b4be8bb355baf1f254"
server_key_iv: "3186c65f55cc9b316cb429409a58dcff"
server_key_version: "encryptionkey@0"
I20260131 03:14:35.150014 10314 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-1/wal/instance:
uuid: "13f76f18a15b4d2e82e79cf574f4f34e"
format_stamp: "Formatted at 2026-01-31 03:14:35 on dist-test-slave-5nnp"
server_key: "5a0c5275f9b1e5b4be8bb355baf1f254"
server_key_iv: "3186c65f55cc9b316cb429409a58dcff"
server_key_version: "encryptionkey@0"
I20260131 03:14:35.151163 10314 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:14:35.151856 10329 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:14:35.152065 10314 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260131 03:14:35.152139 10314 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-1/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-1/wal
uuid: "13f76f18a15b4d2e82e79cf574f4f34e"
format_stamp: "Formatted at 2026-01-31 03:14:35 on dist-test-slave-5nnp"
server_key: "5a0c5275f9b1e5b4be8bb355baf1f254"
server_key_iv: "3186c65f55cc9b316cb429409a58dcff"
server_key_version: "encryptionkey@0"
I20260131 03:14:35.152194 10314 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:14:35.159441 10314 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:14:35.161175 10314 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:14:35.161329 10314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:14:35.165299 10314 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.61:36965
I20260131 03:14:35.165360 10381 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.61:36965 every 8 connection(s)
I20260131 03:14:35.165689 10314 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-1/data/info.pb
I20260131 03:14:35.166427 10382 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:14:35.168378 10382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36721 } has no permanent_uuid. Determining permanent_uuid...
I20260131 03:14:35.170583 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 10314
I20260131 03:14:35.170704 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-1/wal/instance
I20260131 03:14:35.170866 19792 external_mini_cluster.cc:1468] Setting key 7026785fd39bcf9e94a1997f90dbd87e
Jan 31 03:14:35 dist-test-slave-5nnp krb5kdc[10221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829275, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.61@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:14:35.178040 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.169103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37404 (local address 127.19.84.62:36721)
0131 03:14:35.169172 (+ 69us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.169174 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.169367 (+ 193us) server_negotiation.cc:408] Connection header received
0131 03:14:35.169869 (+ 502us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.169870 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.169883 (+ 13us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.169963 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.170954 (+ 991us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.171512 (+ 558us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:35.172309 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.172520 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:35.174803 (+ 2283us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:35.174811 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:35.174811 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:35.174821 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:35.176682 (+ 1861us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.177122 (+ 440us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.177123 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.177124 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.177153 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.177354 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.177355 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.177356 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.177491 (+ 135us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:35.177561 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:35.177821 (+ 260us) server_negotiation.cc:300] Negotiation successful
0131 03:14:35.177926 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260131 03:14:35.178063 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.169063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:36721 (local address 127.0.0.1:37404)
0131 03:14:35.169251 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.169279 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0131 03:14:35.169713 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:14:35.170042 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:14:35.170045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:14:35.170374 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:14:35.170818 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:35.170823 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.171615 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:14:35.171616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:14:35.172209 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:35.172211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.172352 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:35.173066 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:14:35.173115 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:14:35.174698 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:14:35.176791 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:35.176797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:35.176800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:35.177043 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:35.177226 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:35.177227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:35.177227 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:35.177289 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:35.177562 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:14:35.177565 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:14:35.177685 (+ 120us) client_negotiation.cc:769] Sending connection context
0131 03:14:35.177769 (+ 84us) client_negotiation.cc:240] Negotiation successful
0131 03:14:35.177880 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for kudu/127.19.84.60@KRBTEST.COM; defaulting to no policy
I20260131 03:14:35.178800 10382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.84.60@KRBTEST.COM" created.
Jan 31 03:14:35 dist-test-slave-5nnp krb5kdc[10221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829275, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.61@KRBTEST.COM for kudu/127.19.84.61@KRBTEST.COM
I20260131 03:14:35.185891 10386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.179178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33550 (local address 127.19.84.61:36965)
0131 03:14:35.179322 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.179329 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.179340 (+ 11us) server_negotiation.cc:408] Connection header received
0131 03:14:35.179355 (+ 15us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.179356 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.179366 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.179467 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.179802 (+ 335us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.180542 (+ 740us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:35.181079 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.181269 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:35.183049 (+ 1780us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:35.183058 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:35.183058 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:35.183064 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:35.184701 (+ 1637us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.185073 (+ 372us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.185076 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.185076 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.185106 (+ 30us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.185262 (+ 156us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.185263 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.185263 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.185363 (+ 100us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:35.185428 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:35.185645 (+ 217us) server_negotiation.cc:300] Negotiation successful
0131 03:14:35.185780 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260131 03:14:35.185900 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.179091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:36965 (local address 127.0.0.1:33550)
0131 03:14:35.179154 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.179167 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:14:35.179215 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:14:35.179478 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:14:35.179479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:14:35.179600 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:14:35.179732 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:35.179734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.180616 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:14:35.180616 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:14:35.181010 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:35.181011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.181069 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:35.181545 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:14:35.181549 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:14:35.182929 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:14:35.184783 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:35.184784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:35.184785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:35.184985 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:35.185166 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:35.185167 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:35.185167 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:35.185200 (+ 33us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:35.185443 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:14:35.185444 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:14:35.185500 (+ 56us) client_negotiation.cc:769] Sending connection context
0131 03:14:35.185586 (+ 86us) client_negotiation.cc:240] Negotiation successful
0131 03:14:35.185711 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260131 03:14:35.186646 10382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } has no permanent_uuid. Determining permanent_uuid...
I20260131 03:14:35.187018 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.186882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:39859 (local address 127.0.0.1:54908)
0131 03:14:35.186930 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.186999 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260131 03:14:35.187161 10330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.84.60:39859: connect: Connection refused (error 111)
I20260131 03:14:35.187681 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.187596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:39859 (local address 127.0.0.1:54922)
0131 03:14:35.187640 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.187665 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260131 03:14:35.187813 10382 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.60:39859: Network error: Client connection negotiation failed: client connection to 127.19.84.60:39859: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.84.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.19.84.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.19.84.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.84.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.84.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.19.84.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/kudu.keytab.
I20260131 03:14:35.222536 19792 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWVHG5r/build/release/bin/kudu
/tmp/dist-test-taskWVHG5r/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.60:39859
--webserver_interface=127.19.84.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.19.84.20:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.84.62:36721,127.19.84.61:36965,127.19.84.60:39859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260131 03:14:35.227206 10305 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 } attempt: 3
Jan 31 03:14:35 dist-test-slave-5nnp krb5kdc[10221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829274, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for kudu/127.19.84.61@KRBTEST.COM
I20260131 03:14:35.234467 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.227638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:36965 (local address 127.0.0.1:33558)
0131 03:14:35.227704 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.227720 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0131 03:14:35.227796 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:14:35.227962 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:14:35.227963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:14:35.228167 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:14:35.228347 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:35.228349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.229204 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:14:35.229206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:14:35.229567 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:35.229569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.229660 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:35.230265 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:14:35.230270 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:14:35.231590 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:14:35.233522 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:35.233523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:35.233524 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:35.233684 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:35.233864 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:35.233865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:35.233865 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:35.233906 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:35.234197 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:14:35.234198 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:14:35.234243 (+ 45us) client_negotiation.cc:769] Sending connection context
0131 03:14:35.234302 (+ 59us) client_negotiation.cc:240] Negotiation successful
0131 03:14:35.234388 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260131 03:14:35.234573 10386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.227720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33558 (local address 127.19.84.61:36965)
0131 03:14:35.227788 (+ 68us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.227791 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.227804 (+ 13us) server_negotiation.cc:408] Connection header received
0131 03:14:35.227870 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.227871 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.227886 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.227954 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.228450 (+ 496us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.229064 (+ 614us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:35.229655 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.229864 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:35.231686 (+ 1822us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:35.231691 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:35.231692 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:35.231700 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:35.233410 (+ 1710us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.233765 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.233766 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.233766 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.233795 (+ 29us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.233976 (+ 181us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.233977 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.233978 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.234120 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:35.234199 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:35.234349 (+ 150us) server_negotiation.cc:300] Negotiation successful
0131 03:14:35.234450 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260131 03:14:35.235020 10305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } has no permanent_uuid. Determining permanent_uuid...
I20260131 03:14:35.235419 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.235269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:39859 (local address 127.0.0.1:54930)
0131 03:14:35.235317 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.235351 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260131 03:14:35.235719 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.235629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:39859 (local address 127.0.0.1:54946)
0131 03:14:35.235675 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.235699 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260131 03:14:35.235837 10305 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.60:39859: Network error: Client connection negotiation failed: client connection to 127.19.84.60:39859: connect: Connection refused (error 111)
I20260131 03:14:35.237010 10382 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } attempt: 1
I20260131 03:14:35.237381 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.237273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:39859 (local address 127.0.0.1:54958)
0131 03:14:35.237323 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.237354 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260131 03:14:35.237788 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.237674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:39859 (local address 127.0.0.1:54968)
0131 03:14:35.237737 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.237767 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260131 03:14:35.237908 10382 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.60:39859: Network error: Client connection negotiation failed: client connection to 127.19.84.60:39859: connect: Connection refused (error 111)
I20260131 03:14:35.267017 10305 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } attempt: 1
I20260131 03:14:35.267593 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.267444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:39859 (local address 127.0.0.1:54974)
0131 03:14:35.267506 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.267556 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260131 03:14:35.268028 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.267914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:39859 (local address 127.0.0.1:54978)
0131 03:14:35.267972 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.268003 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260131 03:14:35.268162 10305 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.60:39859: Network error: Client connection negotiation failed: client connection to 127.19.84.60:39859: connect: Connection refused (error 111)
I20260131 03:14:35.306072 10382 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } attempt: 2
I20260131 03:14:35.306550 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.306417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:39859 (local address 127.0.0.1:54980)
0131 03:14:35.306480 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.306518 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260131 03:14:35.306902 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.306806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:39859 (local address 127.0.0.1:54984)
0131 03:14:35.306849 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.306873 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.84.60:39859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260131 03:14:35.306998 10382 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.19.84.60:39859: Network error: Client connection negotiation failed: client connection to 127.19.84.60:39859: connect: Connection refused (error 111)
W20260131 03:14:35.307222 10391 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260131 03:14:35.307438 10391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260131 03:14:35.307493 10391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260131 03:14:35.307528 10391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260131 03:14:35.308991 10391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260131 03:14:35.309062 10391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260131 03:14:35.309087 10391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260131 03:14:35.309113 10391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260131 03:14:35.309144 10391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260131 03:14:35.310755 10391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.84.20:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.84.62:36721,127.19.84.61:36965,127.19.84.60:39859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/krb5kdc/kudu.keytab
--principal=kudu/127.19.84.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.84.60:39859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.84.60
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type RELEASE
built by None at 31 Jan 2026 02:43:14 UTC on 5fd53c4cbb9d
build id 10271
I20260131 03:14:35.311012 10391 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260131 03:14:35.311203 10391 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260131 03:14:35.313609 10396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260131 03:14:35.313613 10397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:35.313699 10391 server_base.cc:1060] running on GCE node
W20260131 03:14:35.313830 10399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260131 03:14:35.314040 10391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260131 03:14:35.314294 10391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260131 03:14:35.315438 10391 hybrid_clock.cc:648] HybridClock initialized: now 1769829275315434 us; error 52 us; skew 500 ppm
Jan 31 03:14:35 dist-test-slave-5nnp krb5kdc[10221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829275, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260131 03:14:35.317780 10391 init.cc:377] Logged in from keytab as kudu/127.19.84.60@KRBTEST.COM (short username kudu)
I20260131 03:14:35.318377 10391 webserver.cc:492] Webserver started at http://127.19.84.60:39775/ using document root <none> and password file <none>
I20260131 03:14:35.318574 10391 fs_manager.cc:362] Metadata directory not provided
I20260131 03:14:35.318629 10391 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260131 03:14:35.318763 10391 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260131 03:14:35.319802 10391 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-2/data/instance:
uuid: "3a7e4aa819b74066a65a62399a6d25ab"
format_stamp: "Formatted at 2026-01-31 03:14:35 on dist-test-slave-5nnp"
server_key: "ce9bee434cd4a7e29250ec61bba2fe5c"
server_key_iv: "f6b098df299a4290958eaca349029030"
server_key_version: "encryptionkey@0"
I20260131 03:14:35.320137 10391 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-2/wal/instance:
uuid: "3a7e4aa819b74066a65a62399a6d25ab"
format_stamp: "Formatted at 2026-01-31 03:14:35 on dist-test-slave-5nnp"
server_key: "ce9bee434cd4a7e29250ec61bba2fe5c"
server_key_iv: "f6b098df299a4290958eaca349029030"
server_key_version: "encryptionkey@0"
I20260131 03:14:35.321367 10391 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260131 03:14:35.322065 10406 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260131 03:14:35.322378 10391 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260131 03:14:35.322474 10391 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-2/data,/tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-2/wal
uuid: "3a7e4aa819b74066a65a62399a6d25ab"
format_stamp: "Formatted at 2026-01-31 03:14:35 on dist-test-slave-5nnp"
server_key: "ce9bee434cd4a7e29250ec61bba2fe5c"
server_key_iv: "f6b098df299a4290958eaca349029030"
server_key_version: "encryptionkey@0"
I20260131 03:14:35.322546 10391 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260131 03:14:35.335741 10391 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260131 03:14:35.337572 10391 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260131 03:14:35.337733 10391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260131 03:14:35.341944 10391 rpc_server.cc:307] RPC server started. Bound to: 127.19.84.60:39859
I20260131 03:14:35.342026 10458 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.84.60:39859 every 8 connection(s)
I20260131 03:14:35.342334 10391 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-2/data/info.pb
I20260131 03:14:35.343004 10459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260131 03:14:35.343317 10305 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } attempt: 2
I20260131 03:14:35.345315 10459 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36721 } has no permanent_uuid. Determining permanent_uuid...
I20260131 03:14:35.347136 19792 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWVHG5r/build/release/bin/kudu as pid 10391
I20260131 03:14:35.347262 19792 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWVHG5r/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1769828849314684-19792-0/minicluster-data/master-2/wal/instance
I20260131 03:14:35.347477 19792 external_mini_cluster.cc:1468] Setting key e4b1c46966fe8dc8b87ac64b9188d476
Jan 31 03:14:35 dist-test-slave-5nnp krb5kdc[10221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829274, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.62@KRBTEST.COM for kudu/127.19.84.60@KRBTEST.COM
Jan 31 03:14:35 dist-test-slave-5nnp krb5kdc[10221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829275, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.60@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:14:35.353319 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.343688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:39859 (local address 127.0.0.1:54994)
0131 03:14:35.343745 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.343758 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0131 03:14:35.343810 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:14:35.344910 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:14:35.344912 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:14:35.345193 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:14:35.345369 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:35.345372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.346753 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:14:35.346754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:14:35.347269 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:35.347271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.347349 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:35.347853 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:14:35.347859 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:14:35.349301 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:14:35.351444 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:35.351445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:35.351446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:35.351788 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:35.352448 (+ 660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:35.352449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:35.352449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:35.352481 (+ 32us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:35.352913 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:14:35.352913 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:14:35.352959 (+ 46us) client_negotiation.cc:769] Sending connection context
0131 03:14:35.353107 (+ 148us) client_negotiation.cc:240] Negotiation successful
0131 03:14:35.353209 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260131 03:14:35.353628 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.343850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54994 (local address 127.19.84.60:39859)
0131 03:14:35.344028 (+ 178us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.344038 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.344055 (+ 17us) server_negotiation.cc:408] Connection header received
0131 03:14:35.344723 (+ 668us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.344731 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.344778 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.344922 (+ 144us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.345490 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.346599 (+ 1109us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:35.347439 (+ 840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.348110 (+ 671us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:35.349384 (+ 1274us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:35.349391 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:35.349392 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:35.349398 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:35.351325 (+ 1927us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.351886 (+ 561us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.351889 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.351889 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.351953 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.352562 (+ 609us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.352563 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.352563 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.352834 (+ 271us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:35.352939 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:35.353316 (+ 377us) server_negotiation.cc:300] Negotiation successful
0131 03:14:35.353440 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
I20260131 03:14:35.353881 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.345853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.62:36721 (local address 127.0.0.1:37410)
0131 03:14:35.346043 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.346074 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0131 03:14:35.346145 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:14:35.346289 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:14:35.346291 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:14:35.346675 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:14:35.346866 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:35.346868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.347770 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:14:35.347771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:14:35.348699 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:35.348702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.348765 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:35.349259 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:14:35.349287 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:14:35.351061 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:14:35.352818 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:35.352824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:35.352826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:35.353055 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:35.353247 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:35.353247 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:35.353248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:35.353279 (+ 31us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:35.353513 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:14:35.353519 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:14:35.353573 (+ 54us) client_negotiation.cc:769] Sending connection context
0131 03:14:35.353651 (+ 78us) client_negotiation.cc:240] Negotiation successful
0131 03:14:35.353741 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":122,"threads_started":1}
Jan 31 03:14:35 dist-test-slave-5nnp krb5kdc[10221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.60@KRBTEST.COM
I20260131 03:14:35.354367 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.345919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37410 (local address 127.19.84.62:36721)
0131 03:14:35.345987 (+ 68us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.345989 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.346142 (+ 153us) server_negotiation.cc:408] Connection header received
0131 03:14:35.346202 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.346202 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.346211 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.346261 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.347003 (+ 742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.347654 (+ 651us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:35.348785 (+ 1131us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.348989 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:35.351159 (+ 2170us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:35.351163 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:35.351164 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:35.351171 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:35.352738 (+ 1567us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.353133 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.353134 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.353134 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.353158 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.353348 (+ 190us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.353349 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.353349 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.353454 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:35.353512 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:35.354242 (+ 730us) server_negotiation.cc:300] Negotiation successful
0131 03:14:35.354310 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260131 03:14:35.355468 10305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069: Bootstrap starting.
I20260131 03:14:35.356223 10459 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 } has no permanent_uuid. Determining permanent_uuid...
I20260131 03:14:35.356392 10305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069: Neither blocks nor log segments found. Creating new log.
I20260131 03:14:35.356715 10305 log.cc:826] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069: Log is configured to *not* fsync() on all Append() calls
I20260131 03:14:35.357468 10305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069: No bootstrap required, opened a new log
I20260131 03:14:35.357599 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.347866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55008 (local address 127.19.84.60:39859)
0131 03:14:35.348978 (+ 1112us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.348979 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.349120 (+ 141us) server_negotiation.cc:408] Connection header received
0131 03:14:35.349472 (+ 352us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.349473 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.349481 (+ 8us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.349561 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.350527 (+ 966us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.351217 (+ 690us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:35.351990 (+ 773us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.352194 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:35.354536 (+ 2342us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:35.354540 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:35.354540 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:35.354550 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:35.356126 (+ 1576us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.356755 (+ 629us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.356756 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.356757 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.356781 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.356956 (+ 175us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.356957 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.356957 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.357049 (+ 92us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:35.357179 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:35.357354 (+ 175us) server_negotiation.cc:300] Negotiation successful
0131 03:14:35.357433 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1089,"thread_start_us":71,"threads_started":1}
I20260131 03:14:35.359016 10305 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3601bb0cbc3b480bb69a6906ba48d069" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36721 } } peers { permanent_uuid: "13f76f18a15b4d2e82e79cf574f4f34e" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 } } peers { permanent_uuid: "3a7e4aa819b74066a65a62399a6d25ab" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } }
I20260131 03:14:35.359115 10305 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3601bb0cbc3b480bb69a6906ba48d069, State: Initialized, Role: FOLLOWER
I20260131 03:14:35.359155 10305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3601bb0cbc3b480bb69a6906ba48d069" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36721 } } peers { permanent_uuid: "13f76f18a15b4d2e82e79cf574f4f34e" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 } } peers { permanent_uuid: "3a7e4aa819b74066a65a62399a6d25ab" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } }
I20260131 03:14:35.359438 10305 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069 [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:14:35.359412 10466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3601bb0cbc3b480bb69a6906ba48d069" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36721 } } peers { permanent_uuid: "13f76f18a15b4d2e82e79cf574f4f34e" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 } } peers { permanent_uuid: "3a7e4aa819b74066a65a62399a6d25ab" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } } }
I20260131 03:14:35.359493 10466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069 [sys.catalog]: This master's current role is: FOLLOWER
Jan 31 03:14:35 dist-test-slave-5nnp krb5kdc[10221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829275, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.60@KRBTEST.COM for kudu/127.19.84.61@KRBTEST.COM
W20260131 03:14:35.361485 10477 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:14:35.361994 10477 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:14:35.363142 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.356478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.61:36965 (local address 127.0.0.1:33562)
0131 03:14:35.356521 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.356530 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0131 03:14:35.356563 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:14:35.356705 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:14:35.356706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:14:35.356872 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:14:35.357055 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:35.357057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.358015 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:14:35.358015 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:14:35.358523 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:35.358526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.358580 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:35.358983 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:14:35.358987 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:14:35.360304 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:14:35.362203 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:35.362204 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:35.362204 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:35.362376 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:35.362557 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:35.362558 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:35.362559 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:35.362596 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:35.362856 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:14:35.362857 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:14:35.362894 (+ 37us) client_negotiation.cc:769] Sending connection context
0131 03:14:35.362962 (+ 68us) client_negotiation.cc:240] Negotiation successful
0131 03:14:35.363051 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260131 03:14:35.363183 10386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.356540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33562 (local address 127.19.84.61:36965)
0131 03:14:35.356594 (+ 54us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.356596 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.356604 (+ 8us) server_negotiation.cc:408] Connection header received
0131 03:14:35.356627 (+ 23us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.356628 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.356640 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.356704 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.357227 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.357892 (+ 665us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:35.358592 (+ 700us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.358765 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:35.360380 (+ 1615us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:35.360383 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:35.360384 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:35.360390 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:35.362090 (+ 1700us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.362455 (+ 365us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.362456 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.362457 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.362488 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.362674 (+ 186us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.362675 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.362676 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.362780 (+ 104us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:35.362834 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:35.363007 (+ 173us) server_negotiation.cc:300] Negotiation successful
0131 03:14:35.363096 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260131 03:14:35.363566 10459 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } has no permanent_uuid. Determining permanent_uuid...
Jan 31 03:14:35 dist-test-slave-5nnp krb5kdc[10221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.62@KRBTEST.COM
I20260131 03:14:35.363662 10237 master_runner.cc:186] Error getting master registration for 127.19.84.61:36965: OK, instance_id { permanent_uuid: "13f76f18a15b4d2e82e79cf574f4f34e" instance_seqno: 1769829275164197 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260131 03:14:35.365501 10237 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jan 31 03:14:35 dist-test-slave-5nnp krb5kdc[10221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829275, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.60@KRBTEST.COM for kudu/127.19.84.60@KRBTEST.COM
I20260131 03:14:35.368849 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.359471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37418 (local address 127.19.84.62:36721)
0131 03:14:35.360957 (+ 1486us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.360959 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.360969 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:14:35.360997 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.360997 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.361008 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.361067 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.361789 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.362369 (+ 580us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:35.363623 (+ 1254us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.363786 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:35.365910 (+ 2124us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:35.365913 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:35.365914 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:35.365920 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:35.367821 (+ 1901us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.368174 (+ 353us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.368175 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.368175 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.368199 (+ 24us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.368366 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.368367 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.368368 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.368465 (+ 97us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:35.368614 (+ 149us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:35.368674 (+ 60us) server_negotiation.cc:300] Negotiation successful
0131 03:14:35.368751 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1466}
I20260131 03:14:35.370929 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.363790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:39859 (local address 127.0.0.1:55024)
0131 03:14:35.363896 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.363904 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0131 03:14:35.363956 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:14:35.364102 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:14:35.364103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:14:35.364256 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:14:35.364416 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:35.364418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.365169 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:14:35.365170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:14:35.365630 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:35.365632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.365695 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:35.366190 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:14:35.366194 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:14:35.367453 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:14:35.370027 (+ 2574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:35.370028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:35.370028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:35.370205 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:35.370360 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:35.370361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:35.370361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:35.370387 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:35.370622 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:14:35.370623 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:14:35.370659 (+ 36us) client_negotiation.cc:769] Sending connection context
0131 03:14:35.370725 (+ 66us) client_negotiation.cc:240] Negotiation successful
0131 03:14:35.370822 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260131 03:14:35.370983 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.363849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55024 (local address 127.19.84.60:39859)
0131 03:14:35.363878 (+ 29us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.363879 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.363938 (+ 59us) server_negotiation.cc:408] Connection header received
0131 03:14:35.364012 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.364013 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.364023 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.364069 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.364490 (+ 421us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.365088 (+ 598us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:35.365912 (+ 824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.366088 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:35.367544 (+ 1456us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:35.367548 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:35.367548 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:35.367556 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:35.369931 (+ 2375us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.370260 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.370261 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.370261 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.370288 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.370451 (+ 163us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.370452 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.370452 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.370557 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:35.370631 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:35.370764 (+ 133us) server_negotiation.cc:300] Negotiation successful
0131 03:14:35.370879 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":14}
I20260131 03:14:35.372371 10459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a7e4aa819b74066a65a62399a6d25ab: Bootstrap starting.
I20260131 03:14:35.373021 10459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a7e4aa819b74066a65a62399a6d25ab: Neither blocks nor log segments found. Creating new log.
I20260131 03:14:35.373343 10459 log.cc:826] T 00000000000000000000000000000000 P 3a7e4aa819b74066a65a62399a6d25ab: Log is configured to *not* fsync() on all Append() calls
Jan 31 03:14:35 dist-test-slave-5nnp krb5kdc[10221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.84.61@KRBTEST.COM
I20260131 03:14:35.374174 10459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a7e4aa819b74066a65a62399a6d25ab: No bootstrap required, opened a new log
I20260131 03:14:35.375978 10459 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3a7e4aa819b74066a65a62399a6d25ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3601bb0cbc3b480bb69a6906ba48d069" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36721 } } peers { permanent_uuid: "13f76f18a15b4d2e82e79cf574f4f34e" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 } } peers { permanent_uuid: "3a7e4aa819b74066a65a62399a6d25ab" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } }
I20260131 03:14:35.376113 10459 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3a7e4aa819b74066a65a62399a6d25ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a7e4aa819b74066a65a62399a6d25ab, State: Initialized, Role: FOLLOWER
I20260131 03:14:35.376176 10459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a7e4aa819b74066a65a62399a6d25ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3601bb0cbc3b480bb69a6906ba48d069" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36721 } } peers { permanent_uuid: "13f76f18a15b4d2e82e79cf574f4f34e" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 } } peers { permanent_uuid: "3a7e4aa819b74066a65a62399a6d25ab" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } }
I20260131 03:14:35.376490 10459 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3a7e4aa819b74066a65a62399a6d25ab [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:14:35.376772 10480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a7e4aa819b74066a65a62399a6d25ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3601bb0cbc3b480bb69a6906ba48d069" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36721 } } peers { permanent_uuid: "13f76f18a15b4d2e82e79cf574f4f34e" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 } } peers { permanent_uuid: "3a7e4aa819b74066a65a62399a6d25ab" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } } }
I20260131 03:14:35.376855 10480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a7e4aa819b74066a65a62399a6d25ab [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:14:35.377888 10386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.369419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.19.84.61:36965)
0131 03:14:35.369457 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.369459 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.369466 (+ 7us) server_negotiation.cc:408] Connection header received
0131 03:14:35.369511 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.369512 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.369521 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.369584 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.370245 (+ 661us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.370925 (+ 680us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:35.372026 (+ 1101us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.372205 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:35.374556 (+ 2351us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:35.374560 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:35.374560 (+ 0us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:35.374568 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:35.376330 (+ 1762us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.376692 (+ 362us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.376693 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.376694 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.376722 (+ 28us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.376960 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.376961 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.376961 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.377098 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:35.377251 (+ 153us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:35.377658 (+ 407us) server_negotiation.cc:300] Negotiation successful
0131 03:14:35.377760 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260131 03:14:35.378721 10491 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 3a7e4aa819b74066a65a62399a6d25ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:14:35.378854 10491 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:14:35.379244 19792 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260131 03:14:35.380060 10279 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3601bb0cbc3b480bb69a6906ba48d069"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37418
I20260131 03:14:35.380177 10279 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260131 03:14:35.380228 10279 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069 [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:14:35.381172 10279 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3601bb0cbc3b480bb69a6906ba48d069" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36721 } } peers { permanent_uuid: "13f76f18a15b4d2e82e79cf574f4f34e" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 } } peers { permanent_uuid: "3a7e4aa819b74066a65a62399a6d25ab" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } }
I20260131 03:14:35.381368 10279 leader_election.cc:290] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069 [CANDIDATE]: Term 1 election: Requested vote from peers 13f76f18a15b4d2e82e79cf574f4f34e (127.19.84.61:36965), 3a7e4aa819b74066a65a62399a6d25ab (127.19.84.60:39859)
I20260131 03:14:35.379733 10391 master_runner.cc:186] Error getting master registration for 127.19.84.61:36965: OK, instance_id { permanent_uuid: "13f76f18a15b4d2e82e79cf574f4f34e" instance_seqno: 1769829275164197 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260131 03:14:35.381494 10391 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260131 03:14:35.381911 10433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3601bb0cbc3b480bb69a6906ba48d069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3a7e4aa819b74066a65a62399a6d25ab"
I20260131 03:14:35.382015 10433 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3a7e4aa819b74066a65a62399a6d25ab [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:14:35.382831 10433 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3a7e4aa819b74066a65a62399a6d25ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3601bb0cbc3b480bb69a6906ba48d069 in term 1.
I20260131 03:14:35.383021 10253 leader_election.cc:304] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3601bb0cbc3b480bb69a6906ba48d069, 3a7e4aa819b74066a65a62399a6d25ab; no voters:
I20260131 03:14:35.383150 10466 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069 [term 1 FOLLOWER]: Leader election won for term 1
I20260131 03:14:35.383388 10466 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069 [term 1 LEADER]: Becoming Leader. State: Replica: 3601bb0cbc3b480bb69a6906ba48d069, State: Running, Role: LEADER
I20260131 03:14:35.383495 10466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3601bb0cbc3b480bb69a6906ba48d069" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36721 } } peers { permanent_uuid: "13f76f18a15b4d2e82e79cf574f4f34e" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 } } peers { permanent_uuid: "3a7e4aa819b74066a65a62399a6d25ab" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } }
I20260131 03:14:35.383905 10493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3601bb0cbc3b480bb69a6906ba48d069. Latest consensus state: current_term: 1 leader_uuid: "3601bb0cbc3b480bb69a6906ba48d069" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3601bb0cbc3b480bb69a6906ba48d069" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36721 } } peers { permanent_uuid: "13f76f18a15b4d2e82e79cf574f4f34e" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 } } peers { permanent_uuid: "3a7e4aa819b74066a65a62399a6d25ab" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } } }
I20260131 03:14:35.384011 10493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069 [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:35.384189 10499 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260131 03:14:35.384431 10499 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260131 03:14:35.389360 10433 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3a7e4aa819b74066a65a62399a6d25ab [term 1 FOLLOWER]: Refusing update from remote peer 3601bb0cbc3b480bb69a6906ba48d069: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260131 03:14:35.389675 10493 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a7e4aa819b74066a65a62399a6d25ab" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:14:35.391212 10480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a7e4aa819b74066a65a62399a6d25ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3601bb0cbc3b480bb69a6906ba48d069. Latest consensus state: current_term: 1 leader_uuid: "3601bb0cbc3b480bb69a6906ba48d069" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3601bb0cbc3b480bb69a6906ba48d069" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36721 } } peers { permanent_uuid: "13f76f18a15b4d2e82e79cf574f4f34e" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 } } peers { permanent_uuid: "3a7e4aa819b74066a65a62399a6d25ab" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } } }
I20260131 03:14:35.391317 10480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a7e4aa819b74066a65a62399a6d25ab [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:14:35.396885 10466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3601bb0cbc3b480bb69a6906ba48d069" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3601bb0cbc3b480bb69a6906ba48d069" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36721 } } peers { permanent_uuid: "13f76f18a15b4d2e82e79cf574f4f34e" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 } } peers { permanent_uuid: "3a7e4aa819b74066a65a62399a6d25ab" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } } }
I20260131 03:14:35.396984 10466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069 [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:35.397141 10480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a7e4aa819b74066a65a62399a6d25ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3601bb0cbc3b480bb69a6906ba48d069" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3601bb0cbc3b480bb69a6906ba48d069" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36721 } } peers { permanent_uuid: "13f76f18a15b4d2e82e79cf574f4f34e" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 } } peers { permanent_uuid: "3a7e4aa819b74066a65a62399a6d25ab" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } } }
I20260131 03:14:35.397243 10480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a7e4aa819b74066a65a62399a6d25ab [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:14:35.397260 10501 mvcc.cc:204] Tried to move back new op lower bound from 7249220711992193024 to 7249220711971389440. Current Snapshot: MvccSnapshot[applied={T|T < 7249220711992193024}]
I20260131 03:14:35.397343 10504 mvcc.cc:204] Tried to move back new op lower bound from 7249220711992193024 to 7249220711971389440. Current Snapshot: MvccSnapshot[applied={T|T < 7249220711992193024}]
I20260131 03:14:35.398618 10382 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } attempt: 3
Jan 31 03:14:35 dist-test-slave-5nnp krb5kdc[10221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769829275, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.84.61@KRBTEST.COM for kudu/127.19.84.60@KRBTEST.COM
I20260131 03:14:35.403899 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.387844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55030 (local address 127.19.84.60:39859)
0131 03:14:35.387887 (+ 43us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.387889 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.389858 (+ 1969us) server_negotiation.cc:408] Connection header received
0131 03:14:35.390046 (+ 188us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.390048 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.390059 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.390089 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.391798 (+ 1709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.392525 (+ 727us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:35.393557 (+ 1032us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.393752 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:35.396303 (+ 2551us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:35.396306 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:35.396307 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:35.396315 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:35.398538 (+ 2223us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.402373 (+ 3835us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.402374 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.402374 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.402401 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.403291 (+ 890us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.403293 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.403293 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.403401 (+ 108us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:35.403497 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:35.403705 (+ 208us) server_negotiation.cc:300] Negotiation successful
0131 03:14:35.403804 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260131 03:14:35.403988 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.384316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37424 (local address 127.19.84.62:36721)
0131 03:14:35.384356 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.384358 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.388530 (+ 4172us) server_negotiation.cc:408] Connection header received
0131 03:14:35.388557 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.388558 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.388570 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.388617 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.392957 (+ 4340us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.393597 (+ 640us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:35.396520 (+ 2923us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.396720 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:35.398228 (+ 1508us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:35.398232 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:35.398233 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:35.398241 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:35.399844 (+ 1603us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.402897 (+ 3053us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.402898 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.402899 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.402926 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.403246 (+ 320us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.403247 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.403248 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.403402 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:35.403507 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:35.403815 (+ 308us) server_negotiation.cc:300] Negotiation successful
0131 03:14:35.403894 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260131 03:14:35.404736 10499 catalog_manager.cc:1357] Generated new cluster ID: ac793483b13a406faa806ae01c6bed05
I20260131 03:14:35.404781 10499 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260131 03:14:35.406790 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.399071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.84.60:39859 (local address 127.0.0.1:55036)
0131 03:14:35.399167 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0131 03:14:35.399177 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0131 03:14:35.399215 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0131 03:14:35.399453 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0131 03:14:35.399454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0131 03:14:35.399611 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0131 03:14:35.399798 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:35.399800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.400666 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0131 03:14:35.400667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0131 03:14:35.401236 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0131 03:14:35.401238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.401333 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:35.401723 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0131 03:14:35.401727 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0131 03:14:35.403186 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0131 03:14:35.405908 (+ 2722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:35.405909 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:35.405910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:35.406095 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:35.406290 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0131 03:14:35.406290 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0131 03:14:35.406291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0131 03:14:35.406317 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0131 03:14:35.406524 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0131 03:14:35.406524 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0131 03:14:35.406563 (+ 39us) client_negotiation.cc:769] Sending connection context
0131 03:14:35.406617 (+ 54us) client_negotiation.cc:240] Negotiation successful
0131 03:14:35.406701 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260131 03:14:35.407080 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.399127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55036 (local address 127.19.84.60:39859)
0131 03:14:35.399289 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.399291 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.399298 (+ 7us) server_negotiation.cc:408] Connection header received
0131 03:14:35.399318 (+ 20us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.399318 (+ 0us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.399328 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.399400 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.399887 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.400590 (+ 703us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:35.401336 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.401535 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:35.403990 (+ 2455us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:35.403993 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:35.403994 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:35.404000 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:35.405806 (+ 1806us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.406185 (+ 379us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.406186 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.406187 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.406231 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.406368 (+ 137us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.406368 (+ 0us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.406369 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.406457 (+ 88us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:35.406503 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:35.406924 (+ 421us) server_negotiation.cc:300] Negotiation successful
0131 03:14:35.406999 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260131 03:14:35.408260 10382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13f76f18a15b4d2e82e79cf574f4f34e: Bootstrap starting.
I20260131 03:14:35.409076 10382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13f76f18a15b4d2e82e79cf574f4f34e: Neither blocks nor log segments found. Creating new log.
I20260131 03:14:35.409323 10382 log.cc:826] T 00000000000000000000000000000000 P 13f76f18a15b4d2e82e79cf574f4f34e: Log is configured to *not* fsync() on all Append() calls
I20260131 03:14:35.409977 10382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13f76f18a15b4d2e82e79cf574f4f34e: No bootstrap required, opened a new log
I20260131 03:14:35.411368 10382 raft_consensus.cc:359] T 00000000000000000000000000000000 P 13f76f18a15b4d2e82e79cf574f4f34e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3601bb0cbc3b480bb69a6906ba48d069" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36721 } } peers { permanent_uuid: "13f76f18a15b4d2e82e79cf574f4f34e" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 } } peers { permanent_uuid: "3a7e4aa819b74066a65a62399a6d25ab" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } }
I20260131 03:14:35.411531 10382 raft_consensus.cc:740] T 00000000000000000000000000000000 P 13f76f18a15b4d2e82e79cf574f4f34e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13f76f18a15b4d2e82e79cf574f4f34e, State: Initialized, Role: FOLLOWER
I20260131 03:14:35.411597 10382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13f76f18a15b4d2e82e79cf574f4f34e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3601bb0cbc3b480bb69a6906ba48d069" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36721 } } peers { permanent_uuid: "13f76f18a15b4d2e82e79cf574f4f34e" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 } } peers { permanent_uuid: "3a7e4aa819b74066a65a62399a6d25ab" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } }
I20260131 03:14:35.411870 10382 sys_catalog.cc:565] T 00000000000000000000000000000000 P 13f76f18a15b4d2e82e79cf574f4f34e [sys.catalog]: configured and running, proceeding with master startup.
I20260131 03:14:35.412146 10355 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 13f76f18a15b4d2e82e79cf574f4f34e [term 0 FOLLOWER]: Advancing to term 1
I20260131 03:14:35.413153 10355 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 13f76f18a15b4d2e82e79cf574f4f34e [term 1 FOLLOWER]: Refusing update from remote peer 3601bb0cbc3b480bb69a6906ba48d069: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260131 03:14:35.413345 10335 tablet.cc:2378] T 00000000000000000000000000000000 P 13f76f18a15b4d2e82e79cf574f4f34e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260131 03:14:35.413434 10493 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13f76f18a15b4d2e82e79cf574f4f34e" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260131 03:14:35.413694 10356 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3601bb0cbc3b480bb69a6906ba48d069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "13f76f18a15b4d2e82e79cf574f4f34e"
I20260131 03:14:35.414489 10356 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 13f76f18a15b4d2e82e79cf574f4f34e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3601bb0cbc3b480bb69a6906ba48d069 in term 1.
I20260131 03:14:35.414598 10510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13f76f18a15b4d2e82e79cf574f4f34e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3601bb0cbc3b480bb69a6906ba48d069" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36721 } } peers { permanent_uuid: "13f76f18a15b4d2e82e79cf574f4f34e" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 } } peers { permanent_uuid: "3a7e4aa819b74066a65a62399a6d25ab" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } } }
I20260131 03:14:35.414671 10510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13f76f18a15b4d2e82e79cf574f4f34e [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:14:35.416078 10386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:35.387673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.19.84.61:36965)
0131 03:14:35.387715 (+ 42us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.387717 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.389631 (+ 1914us) server_negotiation.cc:408] Connection header received
0131 03:14:35.389896 (+ 265us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.389897 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.389909 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.389971 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.395014 (+ 5043us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.395747 (+ 733us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:35.396968 (+ 1221us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:35.397148 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:35.402128 (+ 4980us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0131 03:14:35.402131 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0131 03:14:35.402132 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0131 03:14:35.402139 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0131 03:14:35.403899 (+ 1760us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.405974 (+ 2075us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.405975 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.405975 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.406002 (+ 27us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0131 03:14:35.415452 (+ 9450us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0131 03:14:35.415454 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0131 03:14:35.415454 (+ 0us) server_negotiation.cc:968] calling sasl_server_step()
0131 03:14:35.415578 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0131 03:14:35.415655 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:35.415898 (+ 243us) server_negotiation.cc:300] Negotiation successful
0131 03:14:35.415997 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260131 03:14:35.419467 10510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13f76f18a15b4d2e82e79cf574f4f34e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3601bb0cbc3b480bb69a6906ba48d069. Latest consensus state: current_term: 1 leader_uuid: "3601bb0cbc3b480bb69a6906ba48d069" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3601bb0cbc3b480bb69a6906ba48d069" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36721 } } peers { permanent_uuid: "13f76f18a15b4d2e82e79cf574f4f34e" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 } } peers { permanent_uuid: "3a7e4aa819b74066a65a62399a6d25ab" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } } }
I20260131 03:14:35.419552 10510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13f76f18a15b4d2e82e79cf574f4f34e [sys.catalog]: This master's current role is: FOLLOWER
I20260131 03:14:35.419593 10510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13f76f18a15b4d2e82e79cf574f4f34e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3601bb0cbc3b480bb69a6906ba48d069" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3601bb0cbc3b480bb69a6906ba48d069" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36721 } } peers { permanent_uuid: "13f76f18a15b4d2e82e79cf574f4f34e" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 } } peers { permanent_uuid: "3a7e4aa819b74066a65a62399a6d25ab" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } } }
I20260131 03:14:35.419644 10510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13f76f18a15b4d2e82e79cf574f4f34e [sys.catalog]: This master's current role is: FOLLOWER
W20260131 03:14:35.420367 10521 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 13f76f18a15b4d2e82e79cf574f4f34e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260131 03:14:35.420413 10521 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260131 03:14:35.421139 10522 mvcc.cc:204] Tried to move back new op lower bound from 7249220711992193024 to 7249220711971389440. Current Snapshot: MvccSnapshot[applied={T|T < 7249220711992193024}]
I20260131 03:14:35.434234 10507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3601bb0cbc3b480bb69a6906ba48d069" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3601bb0cbc3b480bb69a6906ba48d069" member_type: VOTER last_known_addr { host: "127.19.84.62" port: 36721 } } peers { permanent_uuid: "13f76f18a15b4d2e82e79cf574f4f34e" member_type: VOTER last_known_addr { host: "127.19.84.61" port: 36965 } } peers { permanent_uuid: "3a7e4aa819b74066a65a62399a6d25ab" member_type: VOTER last_known_addr { host: "127.19.84.60" port: 39859 } } }
I20260131 03:14:35.434347 10507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069 [sys.catalog]: This master's current role is: LEADER
I20260131 03:14:35.441381 10499 catalog_manager.cc:1380] Generated new certificate authority record
I20260131 03:14:35.442291 10499 catalog_manager.cc:1514] Loading token signing keys...
I20260131 03:14:35.455060 10499 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 3601bb0cbc3b480bb69a6906ba48d069: Generated new TSK 0
I20260131 03:14:35.455399 10499 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260131 03:14:35.482376 10309 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:35.480579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37436 (local address 127.19.84.62:36721)
0131 03:14:35.480628 (+ 49us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.480631 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.481584 (+ 953us) server_negotiation.cc:408] Connection header received
0131 03:14:35.481691 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.481692 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.481704 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.481735 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.482301 (+ 566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37436: BlockingRecv error: recv got EOF from 127.0.0.1:37436 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260131 03:14:35.483368 10386 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:35.481025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33594 (local address 127.19.84.61:36965)
0131 03:14:35.481065 (+ 40us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.481067 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.482809 (+ 1742us) server_negotiation.cc:408] Connection header received
0131 03:14:35.482909 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.482911 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.482921 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.482951 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.483306 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33594: BlockingRecv error: recv got EOF from 127.0.0.1:33594 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260131 03:14:35.484297 10461 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:35.481345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55038 (local address 127.19.84.60:39859)
0131 03:14:35.481383 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.481385 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.483758 (+ 2373us) server_negotiation.cc:408] Connection header received
0131 03:14:35.483866 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.483868 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.483880 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.483910 (+ 30us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.484244 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55038: BlockingRecv error: recv got EOF from 127.0.0.1:55038 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260131 03:14:35.486709 10386 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:35.485995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33610 (local address 127.19.84.61:36965)
0131 03:14:35.486031 (+ 36us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.486032 (+ 1us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.486211 (+ 179us) server_negotiation.cc:408] Connection header received
0131 03:14:35.486320 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.486321 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.486336 (+ 15us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.486367 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.486663 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33610: BlockingRecv error: recv got EOF from 127.0.0.1:33610 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260131 03:14:35.489850 10386 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:35.489080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33618 (local address 127.19.84.61:36965)
0131 03:14:35.489111 (+ 31us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.489113 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.489322 (+ 209us) server_negotiation.cc:408] Connection header received
0131 03:14:35.489415 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.489416 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.489426 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.489454 (+ 28us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.489809 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33618: BlockingRecv error: recv got EOF from 127.0.0.1:33618 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260131 03:14:35.493917 10386 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:35.493197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.19.84.61:36965)
0131 03:14:35.493231 (+ 34us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.493233 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.493434 (+ 201us) server_negotiation.cc:408] Connection header received
0131 03:14:35.493527 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.493528 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.493537 (+ 9us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.493565 (+ 28us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.493874 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33624: BlockingRecv error: recv got EOF from 127.0.0.1:33624 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260131 03:14:35.500141 10386 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:35.499317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33628 (local address 127.19.84.61:36965)
0131 03:14:35.499356 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.499358 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.499632 (+ 274us) server_negotiation.cc:408] Connection header received
0131 03:14:35.499730 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.499731 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.499741 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.499770 (+ 29us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.500094 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33628: BlockingRecv error: recv got EOF from 127.0.0.1:33628 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260131 03:14:35.510311 10386 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:35.509503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33640 (local address 127.19.84.61:36965)
0131 03:14:35.509542 (+ 39us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.509544 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.509765 (+ 221us) server_negotiation.cc:408] Connection header received
0131 03:14:35.509862 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.509864 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.509874 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.509903 (+ 29us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.510258 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33640: BlockingRecv error: recv got EOF from 127.0.0.1:33640 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260131 03:14:35.528807 10386 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:35.527884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33642 (local address 127.19.84.61:36965)
0131 03:14:35.527928 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.527931 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.528201 (+ 270us) server_negotiation.cc:408] Connection header received
0131 03:14:35.528312 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.528313 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.528324 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.528355 (+ 31us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.528747 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33642: BlockingRecv error: recv got EOF from 127.0.0.1:33642 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260131 03:14:35.563290 10386 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:35.562385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33654 (local address 127.19.84.61:36965)
0131 03:14:35.562429 (+ 44us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.562431 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.562682 (+ 251us) server_negotiation.cc:408] Connection header received
0131 03:14:35.562786 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.562787 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.562799 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.562831 (+ 32us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.563230 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33654: BlockingRecv error: recv got EOF from 127.0.0.1:33654 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260131 03:14:35.629743 10386 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:35.628976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33660 (local address 127.19.84.61:36965)
0131 03:14:35.629071 (+ 95us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.629074 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.629153 (+ 79us) server_negotiation.cc:408] Connection header received
0131 03:14:35.629208 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.629209 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.629219 (+ 10us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.629271 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.629676 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33660: BlockingRecv error: recv got EOF from 127.0.0.1:33660 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260131 03:14:35.759661 10386 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:35.758871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33668 (local address 127.19.84.61:36965)
0131 03:14:35.758940 (+ 69us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:35.758943 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:35.759069 (+ 126us) server_negotiation.cc:408] Connection header received
0131 03:14:35.759120 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:35.759121 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:35.759132 (+ 11us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:35.759183 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:35.759587 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33668: BlockingRecv error: recv got EOF from 127.0.0.1:33668 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260131 03:14:36.017561 10386 negotiation.cc:337] Failed RPC negotiation. Trace:
0131 03:14:36.016853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.19.84.61:36965)
0131 03:14:36.016922 (+ 69us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:36.016924 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:36.016934 (+ 10us) server_negotiation.cc:408] Connection header received
0131 03:14:36.016968 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:36.016969 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:36.016981 (+ 12us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:36.017034 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0131 03:14:36.017477 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33674: BlockingRecv error: recv got EOF from 127.0.0.1:33674 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260131 03:14:36.379278 10491 catalog_manager.cc:1269] Loaded cluster ID: ac793483b13a406faa806ae01c6bed05
I20260131 03:14:36.379391 10491 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 3a7e4aa819b74066a65a62399a6d25ab: loading cluster ID for follower catalog manager: success
I20260131 03:14:36.380300 10491 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 3a7e4aa819b74066a65a62399a6d25ab: acquiring CA information for follower catalog manager: success
I20260131 03:14:36.380625 10491 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 3a7e4aa819b74066a65a62399a6d25ab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260131 03:14:36.420835 10521 catalog_manager.cc:1269] Loaded cluster ID: ac793483b13a406faa806ae01c6bed05
I20260131 03:14:36.420914 10521 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 13f76f18a15b4d2e82e79cf574f4f34e: loading cluster ID for follower catalog manager: success
I20260131 03:14:36.421888 10521 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 13f76f18a15b4d2e82e79cf574f4f34e: acquiring CA information for follower catalog manager: success
I20260131 03:14:36.422219 10521 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 13f76f18a15b4d2e82e79cf574f4f34e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260131 03:14:36.533320 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:36.530838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.19.84.61:36965)
0131 03:14:36.530992 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:36.530994 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:36.531006 (+ 12us) server_negotiation.cc:408] Connection header received
0131 03:14:36.531053 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:36.531054 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:36.531093 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:36.531162 (+ 69us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:14:36.531530 (+ 368us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:36.532051 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:36.532714 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:36.532878 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:36.532924 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:14:36.533027 (+ 103us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:14:36.533101 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:36.533222 (+ 121us) server_negotiation.cc:300] Negotiation successful
0131 03:14:36.533248 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":63,"threads_started":1}
I20260131 03:14:36.537446 10597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:36.535286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55054 (local address 127.19.84.60:39859)
0131 03:14:36.535488 (+ 202us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:36.535490 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:36.535499 (+ 9us) server_negotiation.cc:408] Connection header received
0131 03:14:36.535540 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:36.535541 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:36.535568 (+ 27us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:36.535636 (+ 68us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:14:36.535885 (+ 249us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:36.536377 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:36.536905 (+ 528us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:36.537033 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:36.537071 (+ 38us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:14:36.537164 (+ 93us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:14:36.537230 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:36.537290 (+ 60us) server_negotiation.cc:300] Negotiation successful
0131 03:14:36.537311 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":59,"threads_started":1}
I20260131 03:14:36.541967 10597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:36.539562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55064 (local address 127.19.84.60:39859)
0131 03:14:36.539600 (+ 38us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:36.539602 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:36.539718 (+ 116us) server_negotiation.cc:408] Connection header received
0131 03:14:36.539802 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:36.539803 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:36.539817 (+ 14us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:36.539866 (+ 49us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:14:36.540207 (+ 341us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:36.540654 (+ 447us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:36.541490 (+ 836us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:36.541670 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:36.541694 (+ 24us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:14:36.541725 (+ 31us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:14:36.541777 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:36.541859 (+ 82us) server_negotiation.cc:300] Negotiation successful
0131 03:14:36.541881 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260131 03:14:36.542038 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:36.539269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33694 (local address 127.19.84.61:36965)
0131 03:14:36.539311 (+ 42us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:36.539313 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:36.539516 (+ 203us) server_negotiation.cc:408] Connection header received
0131 03:14:36.539625 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:36.539626 (+ 1us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:36.539644 (+ 18us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:36.539716 (+ 72us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:14:36.540005 (+ 289us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:36.540766 (+ 761us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:36.541489 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:36.541646 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:36.541750 (+ 104us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:14:36.541783 (+ 33us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:14:36.541833 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:36.541919 (+ 86us) server_negotiation.cc:300] Negotiation successful
0131 03:14:36.541954 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260131 03:14:36.542395 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0131 03:14:36.539269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37446 (local address 127.19.84.62:36721)
0131 03:14:36.539415 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0131 03:14:36.539418 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0131 03:14:36.539453 (+ 35us) server_negotiation.cc:408] Connection header received
0131 03:14:36.539621 (+ 168us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0131 03:14:36.539623 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0131 03:14:36.539658 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0131 03:14:36.539732 (+ 74us) server_negotiation.cc:227] Negotiated authn=TOKEN
0131 03:14:36.540125 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:36.540886 (+ 761us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0131 03:14:36.541497 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0131 03:14:36.541670 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0131 03:14:36.541840 (+ 170us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0131 03:14:36.541955 (+ 115us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0131 03:14:36.542023 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0131 03:14:36.542265 (+ 242us) server_negotiation.cc:300] Negotiation successful
0131 03:14:36.542297 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":58,"threads_started":1}
I20260131 03:14:36.544163 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 10237
I20260131 03:14:36.551802 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 10314
I20260131 03:14:36.556986 19792 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWVHG5r/build/release/bin/kudu with pid 10391
2026-01-31T03:14:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3835 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (7729 ms total)
[----------] Global test environment tear-down
[==========] 57 tests from 4 test suites ran. (427255 ms total)
[ PASSED ] 56 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration
1 FAILED TEST